Jul 09 03:59:59.127245 ip-10-0-137-40 systemd[1]: Starting Kubernetes Kubelet... Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.550083 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555090 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555101 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555105 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555109 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555112 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555115 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555118 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.600853 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555121 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555124 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555126 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555129 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555132 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555135 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555137 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555140 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555143 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555146 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555149 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555151 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555154 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555156 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555159 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555163 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555168 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555171 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555173 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555176 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.658129 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555179 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.645684 ip-10-0-137-40 systemd[1]: Started Kubernetes Kubelet. Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555182 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555185 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555187 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555191 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555194 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555197 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555199 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555202 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555205 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555207 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555210 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555213 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555215 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555218 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555221 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555223 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555226 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555229 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.660789 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555231 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555234 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555237 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555240 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555242 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555245 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555248 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555250 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555253 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555256 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555259 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555262 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555264 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555267 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555270 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555273 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555275 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555278 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555281 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555283 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.664036 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555286 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555288 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555292 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555295 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555298 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555300 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555303 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555306 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555309 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555312 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555314 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555317 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555319 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555322 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555325 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555327 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555330 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555333 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555337 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.673643 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555340 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555343 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555346 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555349 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555351 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555354 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555357 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555359 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555362 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555364 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555367 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555369 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555373 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.555377 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556075 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556084 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556088 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556094 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556097 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.675665 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556101 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556104 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556106 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556109 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556112 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556114 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556117 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556120 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556122 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556125 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556127 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556130 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556133 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556136 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556139 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556141 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556144 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556147 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556150 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556152 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.678010 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556155 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556157 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556160 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556163 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556165 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556168 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556171 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556186 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556189 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556192 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556194 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556197 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556200 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556205 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556207 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556212 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556216 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556219 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556222 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.681338 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556225 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556227 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556230 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556233 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556236 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556239 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556241 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556244 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556246 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556249 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556252 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556254 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556256 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556259 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556262 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556265 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556268 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556270 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556273 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556275 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.835268 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556279 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556281 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556284 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556286 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556289 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556291 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556296 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556298 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556301 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556303 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556306 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556309 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556311 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556314 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556316 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556319 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556322 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556325 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556328 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556332 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.848333 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556335 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556337 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556340 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556343 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556345 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556348 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556353 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556357 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556359 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556362 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556365 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556368 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556371 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556374 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556377 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556740 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556746 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556751 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556756 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.850562 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556759 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556762 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556766 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556768 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556771 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556774 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556777 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556780 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556782 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556785 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556788 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556790 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556794 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556797 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556800 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556802 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556805 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556807 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556810 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556812 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.851613 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556815 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556817 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556820 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556822 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556824 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556827 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556830 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556832 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556835 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556837 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556840 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556844 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556846 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556849 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556852 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556854 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556857 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556859 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556862 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556864 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.852745 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556867 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556870 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556872 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556875 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556879 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556882 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556885 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556887 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556889 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556892 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556895 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556897 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556899 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556902 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556904 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556907 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556909 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556912 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556914 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556917 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.853568 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556920 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556922 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556925 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556929 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556931 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556934 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556937 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556939 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556942 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556944 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556967 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556970 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556972 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556975 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556978 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556980 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556984 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556987 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556991 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.854395 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556994 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.556997 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557000 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557002 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557005 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557007 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557010 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557012 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557016 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557019 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557022 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557025 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557027 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557036 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557039 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557042 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557113 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557125 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557131 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557135 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 03:59:59.855223 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557139 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557143 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557147 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557150 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557154 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557157 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557161 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557164 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557167 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557172 2570 flags.go:64] FLAG: --cgroup-root="" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557175 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557177 2570 flags.go:64] FLAG: --client-ca-file="" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557180 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557183 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557187 2570 flags.go:64] FLAG: --cluster-domain="" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557189 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557192 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557195 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557199 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557202 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557205 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557209 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557212 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557214 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 09 03:59:59.855809 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557217 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557220 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557223 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557226 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557231 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557234 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557237 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557240 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557243 2570 flags.go:64] FLAG: --enable-server="true" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557246 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557251 2570 flags.go:64] FLAG: --event-burst="100" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557254 2570 flags.go:64] FLAG: --event-qps="50" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557257 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557259 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557262 2570 flags.go:64] FLAG: --eviction-hard="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557266 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557269 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557273 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557276 2570 flags.go:64] FLAG: --eviction-soft="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557278 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557281 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557284 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557287 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557289 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557292 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 03:59:59.856458 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557295 2570 flags.go:64] FLAG: --feature-gates="" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557298 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557302 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557305 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557308 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557311 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557314 2570 flags.go:64] FLAG: --help="false" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557317 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-40.ec2.internal" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557320 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557323 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557326 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557330 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557333 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557336 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557339 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557342 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557345 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557347 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557351 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557353 2570 flags.go:64] FLAG: --kube-reserved="" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557356 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557359 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557362 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557365 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 03:59:59.857196 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557368 2570 flags.go:64] FLAG: --lock-file="" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557371 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557374 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557377 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557382 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557385 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557388 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557391 2570 flags.go:64] FLAG: --logging-format="text" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557393 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557396 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557399 2570 flags.go:64] FLAG: --manifest-url="" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557402 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557406 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557409 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557413 2570 flags.go:64] FLAG: --max-pods="110" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557416 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557419 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557421 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557424 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557430 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557433 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557436 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557443 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557446 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557448 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 03:59:59.858329 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557451 2570 flags.go:64] FLAG: --pod-cidr="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557454 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557460 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557462 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557465 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557468 2570 flags.go:64] FLAG: --port="10250" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557471 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557476 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e62998a59560594c" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557479 2570 flags.go:64] FLAG: --qos-reserved="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557482 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557485 2570 flags.go:64] FLAG: --register-node="true" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557487 2570 flags.go:64] FLAG: --register-with-taints="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557493 2570 flags.go:64] FLAG: --registry-burst="10" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557496 2570 flags.go:64] FLAG: --registry-qps="5" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557499 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557502 2570 flags.go:64] FLAG: --reserved-memory="" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557505 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557508 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557511 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557514 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557516 2570 flags.go:64] FLAG: --runonce="false" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557519 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557522 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557525 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557528 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 03:59:59.860585 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557530 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557533 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557538 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557541 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557544 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557547 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557550 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557552 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557555 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557558 2570 flags.go:64] FLAG: --system-cgroups="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557561 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557566 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557568 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557571 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557576 2570 flags.go:64] FLAG: --tls-min-version="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557579 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557582 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557585 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557588 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557590 2570 flags.go:64] FLAG: --v="2" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557594 2570 flags.go:64] FLAG: --version="false" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557598 2570 flags.go:64] FLAG: --vmodule="" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557602 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.557605 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557724 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.870464 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557727 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557730 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557733 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557735 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557738 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557741 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557743 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557746 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557748 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557752 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557755 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557758 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557760 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557763 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557765 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557768 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557771 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557773 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557776 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557778 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.872517 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557781 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557785 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557787 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557790 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557792 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557795 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557798 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557800 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557803 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557805 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557808 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557810 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557813 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557815 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557818 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557820 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557823 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557825 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557828 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557831 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.880549 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557833 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557839 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557842 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557847 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557850 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557852 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557855 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557857 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557860 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557862 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557865 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557868 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557870 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557874 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557877 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557879 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557881 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557884 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557887 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557889 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.881386 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557892 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557894 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557897 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557899 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557902 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557904 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557907 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557909 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557912 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557914 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557917 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557919 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557921 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557926 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557929 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557932 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557939 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557942 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557945 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.882205 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557961 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557966 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557970 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557973 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557976 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557978 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557982 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557985 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557989 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557993 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.557997 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558001 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558004 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558007 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558009 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558012 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558014 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558017 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.882842 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.558020 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.559067 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568035 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568049 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568068 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568078 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568134 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568139 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568142 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568145 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568148 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568151 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568153 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568156 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568158 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.883344 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568161 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568164 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568168 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568172 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568175 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568179 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568181 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568184 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568187 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568189 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568193 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568195 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568198 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568201 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568203 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568206 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568208 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568211 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568214 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568216 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.883800 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568222 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568224 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568227 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568230 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568232 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568235 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568238 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568240 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568243 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568246 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568249 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568252 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568254 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568257 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568260 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568262 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568265 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568268 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568271 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.884346 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568274 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568276 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568279 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568281 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568284 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568286 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568289 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568291 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568294 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568296 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568299 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568302 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568304 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568309 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568311 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568314 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568318 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568321 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568324 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.884889 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568327 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568329 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568332 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568335 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568337 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568341 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568344 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568347 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568349 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568352 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568354 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568357 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568360 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568362 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568365 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568368 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568370 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568373 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568375 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568378 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.885424 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568380 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568383 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568386 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568388 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568391 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568393 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568397 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568401 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568404 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568406 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568409 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568412 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568416 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568527 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568532 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:59:59.886059 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568535 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568538 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568540 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568545 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568547 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568550 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568553 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568555 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568558 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568561 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568564 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568568 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568572 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568574 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568577 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568580 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568582 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568585 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:59:59.886469 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568588 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568590 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568593 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568595 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568598 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568602 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568605 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568607 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568610 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568612 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568615 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568618 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568620 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568623 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568625 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568628 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568630 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568634 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568637 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568640 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:59:59.887047 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568642 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568645 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568647 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568651 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568654 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568658 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568661 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568664 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568666 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568669 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568671 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568674 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568676 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568679 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568682 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568684 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568695 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568699 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568702 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568704 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:59:59.887571 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568707 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568710 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568712 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568715 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568718 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568720 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568724 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568727 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568730 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568734 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568737 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568740 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568743 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568746 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568749 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568751 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568754 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568756 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568760 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:59:59.888153 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568762 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568765 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568767 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568770 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568772 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568775 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568778 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568780 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568783 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568786 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568789 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568792 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568794 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568797 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568799 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568802 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568805 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568807 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568810 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:59:59.888706 ip-10-0-137-40 kubenswrapper[2570]: W0709 03:59:59.568812 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568817 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.568909 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.574810 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.576460 2570 server.go:1019] "Starting client certificate rotation" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.576561 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.576597 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.579359 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.582405 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.601516 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.605793 2570 log.go:25] "Validated CRI v1 runtime API" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.608524 2570 log.go:25] "Validated CRI v1 image API" Jul 09 04:00:00.018334 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.609298 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 04:00:00.018868 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.612985 2570 fs.go:135] Filesystem UUIDs: map[15b1b22c-e4c6-46a6-8475-a82bcc3bd5eb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f899f332-2c39-4bcf-97d0-7d9b01f8227f:/dev/nvme0n1p4] Jul 09 04:00:00.018868 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.613003 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.618919 2570 manager.go:216] Machine: {Timestamp:2026-07-09 03:59:59.617510102 +0000 UTC m=+0.371867759 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098975 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c69f6b9460811ccfc1867a29d7b3 SystemUUID:ec28c69f-6b94-6081-1ccf-c1867a29d7b3 BootID:3d0abbd8-9230-4a91-ade6-92431b278ada Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:85:de:ec:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:85:de:ec:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:bb:e5:1c:8f:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.619028 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.619104 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.621116 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.621147 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.622530 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.622540 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.622547 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.623297 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.624977 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.629787 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.629933 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.629976 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.629991 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.630009 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.630022 2570 kubelet.go:398] "Adding apiserver pod source" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.630033 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 04:00:00.019482 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.631328 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.631348 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.634742 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.637061 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.637089 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638391 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638403 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638409 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638415 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638423 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638429 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638438 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638443 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638451 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638457 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638465 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.638471 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.639422 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.639429 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.643857 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.643870 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.643858 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.645041 2570 server.go:1268] "Started kubelet" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.645369 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.645487 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.645550 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.647205 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.647503 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b25w" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.648278 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.650327 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-40.ec2.internal.18c08319e306e4ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-40.ec2.internal,UID:ip-10-0-137-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-40.ec2.internal,},FirstTimestamp:2026-07-09 03:59:59.645017259 +0000 UTC m=+0.399374916,LastTimestamp:2026-07-09 03:59:59.645017259 +0000 UTC m=+0.399374916,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-40.ec2.internal,}" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.653053 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.653979 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.654000 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.654066 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.654656 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.654905 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.654922 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.654912 2570 factory.go:153] Registering CRI-O factory Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655007 2570 factory.go:223] Registration of the crio container factory successfully Jul 09 04:00:00.020099 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655054 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655064 2570 factory.go:55] Registering systemd factory Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655064 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b25w" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655073 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655071 2570 factory.go:223] Registration of the systemd container factory successfully Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655082 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655104 2570 factory.go:103] Registering Raw factory Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655119 2570 manager.go:1196] Started watching for new ooms in manager Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.655414 2570 manager.go:349] Starting recovery of all containers Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.665943 2570 manager.go:354] Recovery completed Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.666754 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.669975 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-40.ec2.internal\" not found" node="ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.671640 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.674250 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.674279 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.674289 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.676007 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.676020 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.676029 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.676039 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.677111 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.677136 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.677150 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.678251 2570 policy_none.go:49] "None policy: Start" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.678269 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.678283 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.679671 2570 policy_none.go:47] "Start" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716512 2570 manager.go:337] "Starting Device Plugin manager" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.716540 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716550 2570 server.go:87] "Starting device plugin registration server" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716769 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716778 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716879 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716971 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.716980 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.717503 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.717650 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.762384 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.763589 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.763629 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.763671 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.763714 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.766375 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.817540 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.818381 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.818414 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.818430 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.818456 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.827898 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.827917 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-40.ec2.internal\": node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.022601 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.851632 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.864763 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal"] Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.865182 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.867063 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.867090 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.867100 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.868282 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.868603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.868623 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869269 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869288 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869297 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869484 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869495 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.869504 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.872381 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.872400 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.873156 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.873177 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.873189 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.890510 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-40.ec2.internal\" not found" node="ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.894700 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-40.ec2.internal\" not found" node="ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: E0709 03:59:59.951775 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.958099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.958122 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.023925 ip-10-0-137-40 kubenswrapper[2570]: I0709 03:59:59.958139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6fcc3f92e7e2909dcce9ec4feecf63-config\") pod \"kube-apiserver-proxy-ip-10-0-137-40.ec2.internal\" (UID: \"2e6fcc3f92e7e2909dcce9ec4feecf63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:00.052248 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6fcc3f92e7e2909dcce9ec4feecf63-config\") pod \"kube-apiserver-proxy-ip-10-0-137-40.ec2.internal\" (UID: \"2e6fcc3f92e7e2909dcce9ec4feecf63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058705 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6fcc3f92e7e2909dcce9ec4feecf63-config\") pod \"kube-apiserver-proxy-ip-10-0-137-40.ec2.internal\" (UID: \"2e6fcc3f92e7e2909dcce9ec4feecf63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058712 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.058735 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa23252013c2c18ed6864d5b52cf68fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal\" (UID: \"fa23252013c2c18ed6864d5b52cf68fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.155452 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:00.153045 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.284334 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.193502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.284334 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.197100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:00.284334 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:00.254114 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:00.424088 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:00.354554 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:01.713073 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:00.454944 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:01.713073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:00.544524 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:01.713073 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:01.706255 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.726521 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.726631 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.726672 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.726673 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:01.806957 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.809516 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 03:55:00 +0000 UTC" deadline="2028-02-23 21:55:04.37537182 +0000 UTC" Jul 09 04:00:01.823516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:01.809549 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14273h55m2.565826372s" Jul 09 04:00:01.996962 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:01.907304 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:02.071882 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.007623 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:02.134048 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.108142 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:02.213372 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.208619 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-40.ec2.internal\" not found" Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:02.235062 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6fcc3f92e7e2909dcce9ec4feecf63.slice/crio-c2d7bc04c969429f60a6af9780469ed32a763e33c62bdaec993242d7d846b0f1 WatchSource:0}: Error finding container c2d7bc04c969429f60a6af9780469ed32a763e33c62bdaec993242d7d846b0f1: Status 404 returned error can't find the container with id c2d7bc04c969429f60a6af9780469ed32a763e33c62bdaec993242d7d846b0f1 Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:02.235541 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa23252013c2c18ed6864d5b52cf68fc.slice/crio-12b3f7c1f053d4e2f1927159d8b6b325cf18c9e51a3163171efea44e569553b1 WatchSource:0}: Error finding container 12b3f7c1f053d4e2f1927159d8b6b325cf18c9e51a3163171efea44e569553b1: Status 404 returned error can't find the container with id 12b3f7c1f053d4e2f1927159d8b6b325cf18c9e51a3163171efea44e569553b1 Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.242501 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.271060 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.304779 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.317021 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.318020 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" Jul 09 04:00:02.328343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.325891 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 04:00:02.689025 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.609698 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.782314 2570 apiserver.go:52] "Watching apiserver" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.786238 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.789302 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.789648 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-tvg2l","openshift-image-registry/node-ca-djx94","openshift-network-diagnostics/network-check-target-vfs25","openshift-dns/node-resolver-78sl6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal","openshift-multus/multus-additional-cni-plugins-6p5qz","openshift-multus/multus-n5d5b","openshift-multus/network-metrics-daemon-xp2dp","openshift-network-operator/iptables-alerter-59bw2","openshift-ovn-kubernetes/ovnkube-node-qqqng","kube-system/global-pull-secret-syncer-54pxr","kube-system/konnectivity-agent-bbtkq","kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4"] Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.792259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.793683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.794249 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.794256 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d88sv\"" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.794813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.795256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.795099 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.796103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.795373 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 04:00:02.796103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.795698 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x8fpb\"" Jul 09 04:00:02.796103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.795724 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.796575 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.796676 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xgrpj\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.796682 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.796806 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.796985 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.797028 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:02.797244 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.797089 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:02.799900 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.798538 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.799900 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.799867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.800131 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.800114 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 04:00:02.800270 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.800250 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7jbbn\"" Jul 09 04:00:02.800270 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.800259 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 04:00:02.801596 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.801577 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ssm2l\"" Jul 09 04:00:02.801776 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.801760 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:02.801841 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.801825 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:02.801908 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.801847 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.801965 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.801951 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.802705 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.802689 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.804001 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.803986 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.804552 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.804522 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.804552 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.804545 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 04:00:02.804697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.804628 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8q8zp\"" Jul 09 04:00:02.804697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.804551 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.805017 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.805000 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 04:00:02.806228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.805791 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 04:00:02.806228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.805965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.807498 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.806760 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 04:00:02.807498 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.806849 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 04:00:02.807498 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.807260 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:02.808659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.808106 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.808659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.808186 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 04:00:02.808659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.808494 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.809028 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.809005 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-928c7\"" Jul 09 04:00:02.810129 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.810102 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:02.812580 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.812561 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 04:00:02.812664 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.812653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.812788 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.812767 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wxmsm\"" Jul 09 04:00:02.812906 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.812873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.813133 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.813114 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.814828 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.814807 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 04:00:02.814828 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.814820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 04:00:02.814987 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.814861 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ftz8r\"" Jul 09 04:00:02.815335 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.815312 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 04:00:02.817388 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.817373 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 04:00:02.818853 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysconfig\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.818914 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-host-slash\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.818914 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-os-release\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.818914 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-socket-dir-parent\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbf5\" (UniqueName: \"kubernetes.io/projected/cf60a3fd-3717-4311-ba96-b89248760807-kube-api-access-cvbf5\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-systemd-units\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bae520-6f0d-47e6-b75c-950c05499b3c-tmp-dir\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.818992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-host\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9hv\" (UniqueName: \"kubernetes.io/projected/49f78cbb-266d-4335-a43e-7fb3423f4f66-kube-api-access-cz9hv\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-iptables-alerter-script\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-binary-copy\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz92\" (UniqueName: \"kubernetes.io/projected/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-kube-api-access-5jz92\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-script-lib\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bae520-6f0d-47e6-b75c-950c05499b3c-hosts-file\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-kubernetes\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-conf\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-log-socket\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-k8s-cni-cncf-io\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-config\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjzx\" (UniqueName: \"kubernetes.io/projected/2fd95163-b562-4bd6-9157-c69ebc28e76b-kube-api-access-tqjzx\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-bin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-kubelet\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819366 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-conf-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-multus-certs\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-etc-kubernetes\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-var-lib-kubelet\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-kubelet-config\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-dbus\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e21662-7592-4c77-80ef-1f8b029160fc-konnectivity-ca\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-netns\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819609 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-system-cni-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-slash\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-system-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-systemd\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-run\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc9wh\" (UniqueName: \"kubernetes.io/projected/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-kube-api-access-xc9wh\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.819799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819731 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-cni-binary-copy\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819755 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovn-node-metrics-cert\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-tuned\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-cnibin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819862 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l4q\" (UniqueName: \"kubernetes.io/projected/d4bae520-6f0d-47e6-b75c-950c05499b3c-kube-api-access-29l4q\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-multus-daemon-config\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819913 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-kubelet\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-lib-modules\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.819984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e21662-7592-4c77-80ef-1f8b029160fc-agent-certs\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-hostroot\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cnibin\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-var-lib-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-node-log\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.820422 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-bin\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820106 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-modprobe-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820164 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-sys\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-tmp\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-netns\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-multus\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-systemd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-etc-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-os-release\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820381 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-ovn\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-netd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-env-overrides\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.821712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.820449 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnflh\" (UniqueName: \"kubernetes.io/projected/9be4d8b2-5274-40cf-b789-f2992ceb9004-kube-api-access-fnflh\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.835909 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.835891 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4l8x" Jul 09 04:00:02.844259 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.844240 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4l8x" Jul 09 04:00:02.905768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.905749 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 04:00:02.921218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-system-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.921314 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-system-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.921602 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-systemd\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.921669 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-run\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.921669 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc9wh\" (UniqueName: \"kubernetes.io/projected/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-kube-api-access-xc9wh\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.921765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-cni-binary-copy\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.921765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.921765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovn-node-metrics-cert\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.921765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-tuned\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.921765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-cnibin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l4q\" (UniqueName: \"kubernetes.io/projected/d4bae520-6f0d-47e6-b75c-950c05499b3c-kube-api-access-29l4q\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-socket-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-registration-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-multus-daemon-config\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-kubelet\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-lib-modules\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e21662-7592-4c77-80ef-1f8b029160fc-agent-certs\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-hostroot\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.922004 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.921991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cnibin\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-var-lib-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-node-log\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-bin\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-modprobe-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-sys\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-tmp\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-netns\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-multus\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922269 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-kubelet\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922320 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-systemd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-etc-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922409 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-hostroot\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-device-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-os-release\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922456 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-lib-modules\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-ovn\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-netd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922540 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-var-lib-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922565 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-run\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-env-overrides\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922585 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-netns\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922618 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-bin\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnflh\" (UniqueName: \"kubernetes.io/projected/9be4d8b2-5274-40cf-b789-f2992ceb9004-kube-api-access-fnflh\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922646 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-multus\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-etc-selinux\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4gb\" (UniqueName: \"kubernetes.io/projected/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kube-api-access-4z4gb\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.923869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysconfig\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-host-slash\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922751 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" event={"ID":"2e6fcc3f92e7e2909dcce9ec4feecf63","Type":"ContainerStarted","Data":"c2d7bc04c969429f60a6af9780469ed32a763e33c62bdaec993242d7d846b0f1"} Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-os-release\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-socket-dir-parent\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbf5\" (UniqueName: \"kubernetes.io/projected/cf60a3fd-3717-4311-ba96-b89248760807-kube-api-access-cvbf5\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-systemd-units\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922872 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-ovn\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bae520-6f0d-47e6-b75c-950c05499b3c-tmp-dir\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-host\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922922 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-cni-netd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9hv\" (UniqueName: \"kubernetes.io/projected/49f78cbb-266d-4335-a43e-7fb3423f4f66-kube-api-access-cz9hv\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922946 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-sys\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-iptables-alerter-script\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-binary-copy\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923003 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-modprobe-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922508 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cnibin\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz92\" (UniqueName: \"kubernetes.io/projected/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-kube-api-access-5jz92\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.924715 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923052 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-systemd-units\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922736 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-script-lib\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923147 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-cni-binary-copy\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bae520-6f0d-47e6-b75c-950c05499b3c-hosts-file\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-kubernetes\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923231 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-host-slash\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-conf\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923256 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-log-socket\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923311 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-k8s-cni-cncf-io\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-config\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjzx\" (UniqueName: \"kubernetes.io/projected/2fd95163-b562-4bd6-9157-c69ebc28e76b-kube-api-access-tqjzx\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-bin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923451 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-kubelet\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923498 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-kubelet\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-conf-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923030 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-d\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-multus-certs\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923535 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf60a3fd-3717-4311-ba96-b89248760807-multus-daemon-config\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923550 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-env-overrides\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923097 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-cnibin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-etc-kubernetes\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.923005 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923603 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-node-log\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923638 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bae520-6f0d-47e6-b75c-950c05499b3c-hosts-file\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-var-lib-kubelet\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-kubelet-config\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-dbus\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.923081 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e21662-7592-4c77-80ef-1f8b029160fc-konnectivity-ca\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.923756 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.423738332 +0000 UTC m=+3.027697051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:02.926997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-netns\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81c5e01e-0e76-493e-9a70-9125d05cd963-host\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923814 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-kubernetes\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-sys-fs\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-system-cni-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-slash\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81c5e01e-0e76-493e-9a70-9125d05cd963-serviceca\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jg2\" (UniqueName: \"kubernetes.io/projected/81c5e01e-0e76-493e-9a70-9125d05cd963-kube-api-access-n9jg2\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923919 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-conf-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923960 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysctl-conf\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923977 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-systemd\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.923987 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-netns\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-log-socket\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-os-release\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924040 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-system-cni-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924054 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-k8s-cni-cncf-io\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.922980 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-run-systemd\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.927818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924105 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.924187 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.424168934 +0000 UTC m=+3.028127661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924224 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e21662-7592-4c77-80ef-1f8b029160fc-konnectivity-ca\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924236 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-socket-dir-parent\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924314 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-etc-openvswitch\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924379 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924425 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-host\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924466 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bae520-6f0d-47e6-b75c-950c05499b3c-tmp-dir\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924553 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9be4d8b2-5274-40cf-b789-f2992ceb9004-host-slash\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924597 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-run-multus-certs\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924640 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-sysconfig\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924647 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" event={"ID":"fa23252013c2c18ed6864d5b52cf68fc","Type":"ContainerStarted","Data":"12b3f7c1f053d4e2f1927159d8b6b325cf18c9e51a3163171efea44e569553b1"} Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924840 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-host-var-lib-cni-bin\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924925 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-iptables-alerter-script\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.924975 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-dbus\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925057 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-etc-kubernetes\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925065 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-script-lib\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.928502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925148 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-multus-cni-dir\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925215 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f78cbb-266d-4335-a43e-7fb3423f4f66-var-lib-kubelet\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925247 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-kubelet-config\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925294 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovnkube-config\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925312 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf60a3fd-3717-4311-ba96-b89248760807-os-release\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925389 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-cni-binary-copy\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.925700 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.926719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-etc-tuned\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.926830 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f78cbb-266d-4335-a43e-7fb3423f4f66-tmp\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.927026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e21662-7592-4c77-80ef-1f8b029160fc-agent-certs\") pod \"konnectivity-agent-bbtkq\" (UID: \"15e21662-7592-4c77-80ef-1f8b029160fc\") " pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.927924 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.927942 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.927956 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:02.928024 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.428009621 +0000 UTC m=+3.031968341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:02.929047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.928570 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9be4d8b2-5274-40cf-b789-f2992ceb9004-ovn-node-metrics-cert\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.929921 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.929893 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc9wh\" (UniqueName: \"kubernetes.io/projected/d1b1bc83-df90-49c5-bd57-62b7ce8972a3-kube-api-access-xc9wh\") pod \"iptables-alerter-59bw2\" (UID: \"d1b1bc83-df90-49c5-bd57-62b7ce8972a3\") " pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:02.937746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.937700 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbf5\" (UniqueName: \"kubernetes.io/projected/cf60a3fd-3717-4311-ba96-b89248760807-kube-api-access-cvbf5\") pod \"multus-n5d5b\" (UID: \"cf60a3fd-3717-4311-ba96-b89248760807\") " pod="openshift-multus/multus-n5d5b" Jul 09 04:00:02.937862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.937837 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz92\" (UniqueName: \"kubernetes.io/projected/e9787e2f-3fde-4bb3-8d4d-a058c3b46757-kube-api-access-5jz92\") pod \"multus-additional-cni-plugins-6p5qz\" (UID: \"e9787e2f-3fde-4bb3-8d4d-a058c3b46757\") " pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:02.938497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.938454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9hv\" (UniqueName: \"kubernetes.io/projected/49f78cbb-266d-4335-a43e-7fb3423f4f66-kube-api-access-cz9hv\") pod \"tuned-tvg2l\" (UID: \"49f78cbb-266d-4335-a43e-7fb3423f4f66\") " pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:02.938497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.938478 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l4q\" (UniqueName: \"kubernetes.io/projected/d4bae520-6f0d-47e6-b75c-950c05499b3c-kube-api-access-29l4q\") pod \"node-resolver-78sl6\" (UID: \"d4bae520-6f0d-47e6-b75c-950c05499b3c\") " pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:02.938957 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.938939 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnflh\" (UniqueName: \"kubernetes.io/projected/9be4d8b2-5274-40cf-b789-f2992ceb9004-kube-api-access-fnflh\") pod \"ovnkube-node-qqqng\" (UID: \"9be4d8b2-5274-40cf-b789-f2992ceb9004\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:02.939122 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:02.939058 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjzx\" (UniqueName: \"kubernetes.io/projected/2fd95163-b562-4bd6-9157-c69ebc28e76b-kube-api-access-tqjzx\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:03.024197 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-device-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-etc-selinux\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4gb\" (UniqueName: \"kubernetes.io/projected/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kube-api-access-4z4gb\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81c5e01e-0e76-493e-9a70-9125d05cd963-host\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-sys-fs\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024291 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-device-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81c5e01e-0e76-493e-9a70-9125d05cd963-serviceca\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.024326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jg2\" (UniqueName: \"kubernetes.io/projected/81c5e01e-0e76-493e-9a70-9125d05cd963-kube-api-access-n9jg2\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024342 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-etc-selinux\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024362 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81c5e01e-0e76-493e-9a70-9125d05cd963-host\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-socket-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-registration-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024553 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024604 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-registration-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024673 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-socket-dir\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024729 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8e03178-1e4c-474e-a4e1-d49fd69a716e-sys-fs\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.024975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.024823 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81c5e01e-0e76-493e-9a70-9125d05cd963-serviceca\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.033537 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.033516 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jg2\" (UniqueName: \"kubernetes.io/projected/81c5e01e-0e76-493e-9a70-9125d05cd963-kube-api-access-n9jg2\") pod \"node-ca-djx94\" (UID: \"81c5e01e-0e76-493e-9a70-9125d05cd963\") " pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.033679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.033656 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4gb\" (UniqueName: \"kubernetes.io/projected/f8e03178-1e4c-474e-a4e1-d49fd69a716e-kube-api-access-4z4gb\") pod \"aws-ebs-csi-driver-node-57qv4\" (UID: \"f8e03178-1e4c-474e-a4e1-d49fd69a716e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.104623 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.104555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78sl6" Jul 09 04:00:03.113964 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.113942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:03.121443 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.121419 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5d5b" Jul 09 04:00:03.124702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.124683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" Jul 09 04:00:03.130060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.130042 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" Jul 09 04:00:03.131921 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.131898 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf60a3fd_3717_4311_ba96_b89248760807.slice/crio-9543452129f7e71d6bccb21d00f504bc0108ece44de1921108b5ff1ac168d9a3 WatchSource:0}: Error finding container 9543452129f7e71d6bccb21d00f504bc0108ece44de1921108b5ff1ac168d9a3: Status 404 returned error can't find the container with id 9543452129f7e71d6bccb21d00f504bc0108ece44de1921108b5ff1ac168d9a3 Jul 09 04:00:03.141265 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.141241 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f78cbb_266d_4335_a43e_7fb3423f4f66.slice/crio-442234b7a099e8948e3013d9a1469248e568d0f08d8b5083bc56b1a0978cb786 WatchSource:0}: Error finding container 442234b7a099e8948e3013d9a1469248e568d0f08d8b5083bc56b1a0978cb786: Status 404 returned error can't find the container with id 442234b7a099e8948e3013d9a1469248e568d0f08d8b5083bc56b1a0978cb786 Jul 09 04:00:03.142666 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.142648 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59bw2" Jul 09 04:00:03.148869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.148852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:03.151131 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.151107 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b1bc83_df90_49c5_bd57_62b7ce8972a3.slice/crio-b2549177dd3b9b806197128597ae4349d1bb76c887b45d39e50522cf2ccb96bd WatchSource:0}: Error finding container b2549177dd3b9b806197128597ae4349d1bb76c887b45d39e50522cf2ccb96bd: Status 404 returned error can't find the container with id b2549177dd3b9b806197128597ae4349d1bb76c887b45d39e50522cf2ccb96bd Jul 09 04:00:03.154340 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.153654 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-djx94" Jul 09 04:00:03.158839 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.158815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" Jul 09 04:00:03.160062 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.160042 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be4d8b2_5274_40cf_b789_f2992ceb9004.slice/crio-4279ce134428326fcfdd3526d1b8d6c593c44884cc12c10ac31c1decc5ec6a37 WatchSource:0}: Error finding container 4279ce134428326fcfdd3526d1b8d6c593c44884cc12c10ac31c1decc5ec6a37: Status 404 returned error can't find the container with id 4279ce134428326fcfdd3526d1b8d6c593c44884cc12c10ac31c1decc5ec6a37 Jul 09 04:00:03.163236 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.163217 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c5e01e_0e76_493e_9a70_9125d05cd963.slice/crio-58a1cfd6ca4460d7d6a68a5a23e495689b6b087a9048374ce1cb596aca5ec44c WatchSource:0}: Error finding container 58a1cfd6ca4460d7d6a68a5a23e495689b6b087a9048374ce1cb596aca5ec44c: Status 404 returned error can't find the container with id 58a1cfd6ca4460d7d6a68a5a23e495689b6b087a9048374ce1cb596aca5ec44c Jul 09 04:00:03.168201 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:03.168177 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e03178_1e4c_474e_a4e1_d49fd69a716e.slice/crio-0a20de46284d7a9a47e48124b1fa8c5d6b19c734d5ab8e3ab21661ec81cb8abf WatchSource:0}: Error finding container 0a20de46284d7a9a47e48124b1fa8c5d6b19c734d5ab8e3ab21661ec81cb8abf: Status 404 returned error can't find the container with id 0a20de46284d7a9a47e48124b1fa8c5d6b19c734d5ab8e3ab21661ec81cb8abf Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.427108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.427205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.427321 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.427393 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:04.427370862 +0000 UTC m=+4.031329582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.427775 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:03.427900 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.427828 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:04.427812282 +0000 UTC m=+4.031771006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:03.528984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.528420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:03.528984 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.528578 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:03.528984 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.528599 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:03.528984 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.528612 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:03.528984 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.528665 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:04.528645942 +0000 UTC m=+4.132604681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:03.846419 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.846284 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:55:02 +0000 UTC" deadline="2028-04-01 13:48:39.449563102 +0000 UTC" Jul 09 04:00:03.846419 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.846383 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15177h48m35.603191408s" Jul 09 04:00:03.915046 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.915010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:03.915226 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.915156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:03.915609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.915586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:03.915814 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:03.915758 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:03.928895 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.928864 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" event={"ID":"f8e03178-1e4c-474e-a4e1-d49fd69a716e","Type":"ContainerStarted","Data":"0a20de46284d7a9a47e48124b1fa8c5d6b19c734d5ab8e3ab21661ec81cb8abf"} Jul 09 04:00:03.933182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.933156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-djx94" event={"ID":"81c5e01e-0e76-493e-9a70-9125d05cd963","Type":"ContainerStarted","Data":"58a1cfd6ca4460d7d6a68a5a23e495689b6b087a9048374ce1cb596aca5ec44c"} Jul 09 04:00:03.936466 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.936391 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59bw2" event={"ID":"d1b1bc83-df90-49c5-bd57-62b7ce8972a3","Type":"ContainerStarted","Data":"b2549177dd3b9b806197128597ae4349d1bb76c887b45d39e50522cf2ccb96bd"} Jul 09 04:00:03.939928 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.939851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerStarted","Data":"3879039eb7215de9732673e0825de64163e0a1c81b3c8a082b0d62bbba6d0c1c"} Jul 09 04:00:03.942063 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.941993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5d5b" event={"ID":"cf60a3fd-3717-4311-ba96-b89248760807","Type":"ContainerStarted","Data":"9543452129f7e71d6bccb21d00f504bc0108ece44de1921108b5ff1ac168d9a3"} Jul 09 04:00:03.943818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.943767 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78sl6" event={"ID":"d4bae520-6f0d-47e6-b75c-950c05499b3c","Type":"ContainerStarted","Data":"962dd0eef6e33e5ffa8f6a3c352c3d1801e97730cf9ca60069b8585aefd03879"} Jul 09 04:00:03.946437 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.946262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"4279ce134428326fcfdd3526d1b8d6c593c44884cc12c10ac31c1decc5ec6a37"} Jul 09 04:00:03.949284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.949262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" event={"ID":"49f78cbb-266d-4335-a43e-7fb3423f4f66","Type":"ContainerStarted","Data":"442234b7a099e8948e3013d9a1469248e568d0f08d8b5083bc56b1a0978cb786"} Jul 09 04:00:03.951926 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:03.951904 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bbtkq" event={"ID":"15e21662-7592-4c77-80ef-1f8b029160fc","Type":"ContainerStarted","Data":"9086a1a6907a4725f60e4abfe57beda9413344726b54ae40eef17977c7606079"} Jul 09 04:00:04.435499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.435414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:04.435671 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.435518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:04.435671 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.435664 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:04.435788 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.435728 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:06.435707565 +0000 UTC m=+6.039666287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:04.436151 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.436133 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:04.436228 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.436188 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:06.436173147 +0000 UTC m=+6.040131869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:04.536578 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.536512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:04.536737 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.536700 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:04.536737 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.536725 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:04.536840 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.536739 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:04.536840 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.536799 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:06.536779909 +0000 UTC m=+6.140738629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:04.599602 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.599570 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:04.847193 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.847101 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:55:02 +0000 UTC" deadline="2028-02-09 20:55:31.76230948 +0000 UTC" Jul 09 04:00:04.847193 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.847140 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h55m26.915174252s" Jul 09 04:00:04.915222 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:04.915191 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:04.915467 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:04.915415 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:05.917202 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:05.916327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:05.917202 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:05.916707 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:05.917202 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:05.916527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:05.917202 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:05.917150 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:05.971526 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:05.971488 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" event={"ID":"2e6fcc3f92e7e2909dcce9ec4feecf63","Type":"ContainerStarted","Data":"569d43c31daec5e2f9c34ba23ffd9868dd0e3bbad9d91e6fcd4f2c886e568ba5"} Jul 09 04:00:06.452686 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.452644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:06.452861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.452700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:06.452861 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.452811 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:06.452993 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.452860 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:06.452993 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.452892 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:10.452866796 +0000 UTC m=+10.056825519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:06.452993 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.452915 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:10.452904297 +0000 UTC m=+10.056863020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:06.553499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.553399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:06.553675 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.553588 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:06.553675 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.553608 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:06.553675 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.553620 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:06.553840 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.553680 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:10.553660906 +0000 UTC m=+10.157619645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:06.915566 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.915472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:06.915730 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:06.915592 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:06.978017 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.977952 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" event={"ID":"fa23252013c2c18ed6864d5b52cf68fc","Type":"ContainerStarted","Data":"2cac1d2e73c9db4943226b903f9eb269fddc080b5f4788ddfa156cfb05fe9864"} Jul 09 04:00:06.992703 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:06.992110 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-40.ec2.internal" podStartSLOduration=4.992089783 podStartE2EDuration="4.992089783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:05.988184525 +0000 UTC m=+5.592143292" watchObservedRunningTime="2026-07-09 04:00:06.992089783 +0000 UTC m=+6.596048526" Jul 09 04:00:07.915177 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:07.915144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:07.915340 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:07.915281 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:07.915648 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:07.915628 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:07.915789 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:07.915746 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:08.914606 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:08.914569 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:08.915030 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:08.914715 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:08.982698 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:08.982664 2570 generic.go:356] "Generic (PLEG): container finished" podID="fa23252013c2c18ed6864d5b52cf68fc" containerID="2cac1d2e73c9db4943226b903f9eb269fddc080b5f4788ddfa156cfb05fe9864" exitCode=0 Jul 09 04:00:08.982698 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:08.982693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" event={"ID":"fa23252013c2c18ed6864d5b52cf68fc","Type":"ContainerDied","Data":"2cac1d2e73c9db4943226b903f9eb269fddc080b5f4788ddfa156cfb05fe9864"} Jul 09 04:00:09.914635 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:09.914601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:09.915140 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:09.914739 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:09.915140 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:09.914858 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:09.915140 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:09.914988 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:10.484988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:10.484881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:10.484988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:10.484946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:10.485226 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.485051 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:10.485226 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.485107 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:10.485226 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.485121 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:18.485099982 +0000 UTC m=+18.089058704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:10.485226 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.485157 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:18.485141824 +0000 UTC m=+18.089100546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:10.586079 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:10.586029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:10.586257 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.586185 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:10.586257 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.586203 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:10.586257 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.586215 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:10.586430 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.586275 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:18.586257373 +0000 UTC m=+18.190216096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:10.916105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:10.916030 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:10.916636 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:10.916139 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:11.914695 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:11.914655 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:11.914874 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:11.914789 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:11.915196 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:11.915167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:11.915322 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:11.915285 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:12.914623 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:12.914573 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:12.915136 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:12.914709 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:13.914384 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:13.914335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:13.914487 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:13.914449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:13.914538 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:13.914514 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:13.914634 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:13.914615 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:14.914800 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:14.914764 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:14.915259 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:14.914897 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:15.914655 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:15.914617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:15.914845 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:15.914756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:15.914845 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:15.914810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:15.915220 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:15.914940 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:16.914996 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:16.914958 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:16.915478 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:16.915126 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:17.914579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:17.914538 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:17.914797 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:17.914539 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:17.914797 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:17.914679 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:17.914797 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:17.914722 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:18.544233 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:18.544197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:18.544233 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:18.544241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:18.544671 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.544344 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:18.544671 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.544378 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:18.544671 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.544438 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:34.544417064 +0000 UTC m=+34.148375794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:18.544671 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.544462 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:00:34.544449909 +0000 UTC m=+34.148408632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:18.645110 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:18.645068 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:18.645257 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.645237 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:18.645302 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.645262 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:18.645302 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.645277 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:18.645390 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.645331 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:34.645317751 +0000 UTC m=+34.249276470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:18.915134 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:18.915040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:18.915291 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:18.915170 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:19.914558 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:19.914521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:19.914558 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:19.914550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:19.915046 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:19.914653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:19.915046 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:19.914802 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:20.915584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:20.915549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:20.916013 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:20.915661 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:21.915030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:21.914992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:21.915204 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:21.914998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:21.915204 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:21.915128 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:21.915311 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:21.915210 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:22.914652 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:22.914345 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:22.915293 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:22.914747 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:23.011157 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011128 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"d2993e302e87098f011b0949d7e9c6386068599892863a012b63ff253aeb3102"} Jul 09 04:00:23.011249 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"4d8027d95d3924d2526f1493ce83642e7f56e24a1bcb37294e0cbb2a7d9c915d"} Jul 09 04:00:23.011249 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"d27d489070f5ced96f1bac82c564b6adfab7b6f926c89f95a4f6b7b9996b371b"} Jul 09 04:00:23.011342 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"f5ec06bd09047c32eff7152e66ad470b22e242e7258fd18f05e906a831e93ed2"} Jul 09 04:00:23.011342 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"861567c7c52ee9dd261b83cc97817f0a22dd172e103d71949629dc8561b9583a"} Jul 09 04:00:23.011342 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.011337 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"b1317408b39e5a7d06d6f0f3b90922cbbafb71865d7c00efc7b88d7a7c6af5ec"} Jul 09 04:00:23.012509 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.012488 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" event={"ID":"49f78cbb-266d-4335-a43e-7fb3423f4f66","Type":"ContainerStarted","Data":"0827795a8158af95a345a84796bd6a2bc8e79561cb680015c9d1f7654f747733"} Jul 09 04:00:23.013895 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.013835 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bbtkq" event={"ID":"15e21662-7592-4c77-80ef-1f8b029160fc","Type":"ContainerStarted","Data":"7c08eb6c17819d5339aa5594c8c32ae589c274dcad6c746406f9c21679d50020"} Jul 09 04:00:23.015538 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.015521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" event={"ID":"fa23252013c2c18ed6864d5b52cf68fc","Type":"ContainerStarted","Data":"f19392288892c1c51d18eef5f661dc2bc6fbe962036b58d9ff4f4f7dadaaed37"} Jul 09 04:00:23.016779 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.016761 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" event={"ID":"f8e03178-1e4c-474e-a4e1-d49fd69a716e","Type":"ContainerStarted","Data":"a93db6fc2c29247b6c91a1d04658962dca24f0a818f1f598e51319667c99c7ef"} Jul 09 04:00:23.017940 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.017921 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-djx94" event={"ID":"81c5e01e-0e76-493e-9a70-9125d05cd963","Type":"ContainerStarted","Data":"2189aa365e16fe8de574b411b325fced4366f31e8df9d885e114c81126ad6928"} Jul 09 04:00:23.019272 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.019254 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="f30ca0b7301e5d758e5b293075a0b266204805f6b457941f9550a6ca73cd7515" exitCode=0 Jul 09 04:00:23.019359 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.019313 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"f30ca0b7301e5d758e5b293075a0b266204805f6b457941f9550a6ca73cd7515"} Jul 09 04:00:23.020706 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.020689 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5d5b" event={"ID":"cf60a3fd-3717-4311-ba96-b89248760807","Type":"ContainerStarted","Data":"ad806687ece87506ccbd4f363f0841ef8a45b07ebe6d6571d6fc069a997b10c9"} Jul 09 04:00:23.022012 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.021993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78sl6" event={"ID":"d4bae520-6f0d-47e6-b75c-950c05499b3c","Type":"ContainerStarted","Data":"5e55238fac91c2c15990fe0b65363fa2449c3f0178d1d8b3239b5cac6c1b4dba"} Jul 09 04:00:23.026941 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.026906 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tvg2l" podStartSLOduration=2.8830663469999998 podStartE2EDuration="22.026895633s" totalImagesPullingTime="19.143829286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.026839104 +0000 UTC m=+22.630797864" watchObservedRunningTime="2026-07-09 04:00:23.026895633 +0000 UTC m=+22.630854411" Jul 09 04:00:23.069665 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.069622 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-40.ec2.internal" podStartSLOduration=21.069605671 podStartE2EDuration="21.069605671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.069590929 +0000 UTC m=+22.673549671" watchObservedRunningTime="2026-07-09 04:00:23.069605671 +0000 UTC m=+22.673564413" Jul 09 04:00:23.086690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.086641 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-78sl6" podStartSLOduration=3.9189665380000003 podStartE2EDuration="23.086628297s" totalImagesPullingTime="19.167661759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.086165923 +0000 UTC m=+22.690124664" watchObservedRunningTime="2026-07-09 04:00:23.086628297 +0000 UTC m=+22.690587038" Jul 09 04:00:23.116057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.116024 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n5d5b" podStartSLOduration=2.933321519 podStartE2EDuration="22.116010947s" totalImagesPullingTime="19.182689428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.115862111 +0000 UTC m=+22.719820851" watchObservedRunningTime="2026-07-09 04:00:23.116010947 +0000 UTC m=+22.719969687" Jul 09 04:00:23.116433 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.116402 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bbtkq" podStartSLOduration=3.9584439529999997 podStartE2EDuration="23.116392893s" totalImagesPullingTime="19.15794894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.100339118 +0000 UTC m=+22.704297858" watchObservedRunningTime="2026-07-09 04:00:23.116392893 +0000 UTC m=+22.720351634" Jul 09 04:00:23.134485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.134450 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-djx94" podStartSLOduration=7.157908074 podStartE2EDuration="22.134439112s" totalImagesPullingTime="14.976531038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.134329904 +0000 UTC m=+22.738288642" watchObservedRunningTime="2026-07-09 04:00:23.134439112 +0000 UTC m=+22.738397854" Jul 09 04:00:23.915174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.915144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:23.915553 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.915147 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:23.915553 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:23.915273 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:23.915553 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:23.915365 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:23.975377 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:23.975336 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 04:00:24.026761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.026543 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" event={"ID":"f8e03178-1e4c-474e-a4e1-d49fd69a716e","Type":"ContainerStarted","Data":"499676c0c92189d7498911b965c58c307f139d73aac81ce9faa4153d9b9eb803"} Jul 09 04:00:24.028069 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.028041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59bw2" event={"ID":"d1b1bc83-df90-49c5-bd57-62b7ce8972a3","Type":"ContainerStarted","Data":"111c4d228d162ab26d3fd0da8c4dde049edcb8c5b4f693a362a1b2872ca66aa2"} Jul 09 04:00:24.043184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.043138 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-59bw2" podStartSLOduration=3.912565104 podStartE2EDuration="23.043121707s" totalImagesPullingTime="19.130556603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:24.042673237 +0000 UTC m=+23.646631981" watchObservedRunningTime="2026-07-09 04:00:24.043121707 +0000 UTC m=+23.647080460" Jul 09 04:00:24.444596 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.444563 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:24.445129 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.445110 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:24.881057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.880938 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T04:00:23.975371299Z","UUID":"2c80f93c-4788-491d-9d0e-ecce6b567a74","Handler":null,"Name":"","Endpoint":""} Jul 09 04:00:24.882360 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.882335 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 04:00:24.882488 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.882389 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 04:00:24.915023 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:24.914996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:24.915176 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:24.915119 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:25.030905 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:25.030873 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:25.031432 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:25.031411 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bbtkq" Jul 09 04:00:25.914435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:25.914401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:25.914713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:25.914405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:25.914713 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:25.914528 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:25.914713 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:25.914623 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:26.036199 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:26.036164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"dac2a811b18cbe25cc7f062a59e358793ad75b4cedd94891f30291c6223b8ca8"} Jul 09 04:00:26.914368 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:26.914321 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:26.914551 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:26.914449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:27.914547 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:27.914330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:27.915259 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:27.914330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:27.915259 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:27.914641 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:27.915259 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:27.914703 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:28.041775 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.041744 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" event={"ID":"f8e03178-1e4c-474e-a4e1-d49fd69a716e","Type":"ContainerStarted","Data":"0edba689e1a0929a7913e384e14efc675068d70092bbe7218e732e0bb2793fa8"} Jul 09 04:00:28.043478 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.043454 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="abcda0d3b03e228ee6a1bd83e250cc5d737a30ad47645cb58dcaba2c0f492129" exitCode=0 Jul 09 04:00:28.043592 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.043531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"abcda0d3b03e228ee6a1bd83e250cc5d737a30ad47645cb58dcaba2c0f492129"} Jul 09 04:00:28.047609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.047587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" event={"ID":"9be4d8b2-5274-40cf-b789-f2992ceb9004","Type":"ContainerStarted","Data":"b206b9973fdde4c9bb9941688e2e2456da6bf085ccf1915a219104bb922dcb43"} Jul 09 04:00:28.047893 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.047870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:28.047983 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.047944 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:28.048031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.047994 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:28.058557 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.058518 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57qv4" podStartSLOduration=3.598823759 podStartE2EDuration="27.058504026s" totalImagesPullingTime="23.459680267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:28.057739973 +0000 UTC m=+27.661698714" watchObservedRunningTime="2026-07-09 04:00:28.058504026 +0000 UTC m=+27.662462761" Jul 09 04:00:28.064964 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.064939 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:28.065153 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.065139 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:00:28.107786 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.107740 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" podStartSLOduration=7.890942992 podStartE2EDuration="27.1077252s" totalImagesPullingTime="19.216782208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:28.107257572 +0000 UTC m=+27.711216332" watchObservedRunningTime="2026-07-09 04:00:28.1077252 +0000 UTC m=+27.711683940" Jul 09 04:00:28.914768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:28.914740 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:28.915312 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:28.914842 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:29.565693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.565611 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfs25"] Jul 09 04:00:29.565826 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.565743 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:29.565880 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:29.565832 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:29.568773 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.568744 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-54pxr"] Jul 09 04:00:29.568889 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.568837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:29.568993 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:29.568970 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:29.569611 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.569585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xp2dp"] Jul 09 04:00:29.569729 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:29.569694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:29.569809 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:29.569791 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:30.915332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:30.915156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:30.915764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:30.915231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:30.915764 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:30.915440 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:30.915764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:30.915254 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:30.915764 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:30.915489 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:30.915764 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:30.915553 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:32.058001 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:32.057972 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="f54940dbcf5b7df2d1eb5786c2fccf048ac8afd227f97042f0fbe7c36050e5be" exitCode=0 Jul 09 04:00:32.058541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:32.058031 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"f54940dbcf5b7df2d1eb5786c2fccf048ac8afd227f97042f0fbe7c36050e5be"} Jul 09 04:00:32.914936 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:32.914907 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:32.915089 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:32.914913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:32.915089 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:32.915007 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:32.915089 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:32.914916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:32.915235 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:32.915089 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:32.915235 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:32.915182 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:34.064553 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.064346 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="a55046bde0f0178cddf64336875279edb892621fe9326e2066c67cc8cabbadfb" exitCode=0 Jul 09 04:00:34.064954 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.064443 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"a55046bde0f0178cddf64336875279edb892621fe9326e2066c67cc8cabbadfb"} Jul 09 04:00:34.557580 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.557545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:34.557727 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.557591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:34.557727 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.557693 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:34.557727 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.557709 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:34.557887 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.557767 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs podName:2fd95163-b562-4bd6-9157-c69ebc28e76b nodeName:}" failed. No retries permitted until 2026-07-09 04:01:06.557749364 +0000 UTC m=+66.161708144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs") pod "network-metrics-daemon-xp2dp" (UID: "2fd95163-b562-4bd6-9157-c69ebc28e76b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:34.557887 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.557787 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret podName:21f4e347-8cf3-4ecc-b6c1-18e9c590e662 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:06.557777179 +0000 UTC m=+66.161735901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret") pod "global-pull-secret-syncer-54pxr" (UID: "21f4e347-8cf3-4ecc-b6c1-18e9c590e662") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:34.657945 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.657916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:34.658096 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.658040 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:34.658096 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.658061 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:34.658096 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.658070 2570 projected.go:196] Error preparing data for projected volume kube-api-access-rj4vk for pod openshift-network-diagnostics/network-check-target-vfs25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:34.658218 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.658126 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk podName:7317d19d-eafb-4286-aec8-896345117a22 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:06.658108674 +0000 UTC m=+66.262067410 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj4vk" (UniqueName: "kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk") pod "network-check-target-vfs25" (UID: "7317d19d-eafb-4286-aec8-896345117a22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:34.915040 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.914995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:34.915174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.914995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:34.915174 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.915115 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54pxr" podUID="21f4e347-8cf3-4ecc-b6c1-18e9c590e662" Jul 09 04:00:34.915174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:34.915138 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:34.915341 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.915243 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp2dp" podUID="2fd95163-b562-4bd6-9157-c69ebc28e76b" Jul 09 04:00:34.915341 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:34.915311 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfs25" podUID="7317d19d-eafb-4286-aec8-896345117a22" Jul 09 04:00:35.210023 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.209951 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-40.ec2.internal" event="NodeReady" Jul 09 04:00:35.210432 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.210081 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 04:00:35.243959 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.243864 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59497797c7-88275"] Jul 09 04:00:35.246100 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.246077 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.248866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.248833 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 04:00:35.249436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.249416 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.249629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.249614 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:00:35.252588 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.252568 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms"] Jul 09 04:00:35.252720 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.252678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.254220 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.254185 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw"] Jul 09 04:00:35.254775 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.254757 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.255438 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.255722 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m"] Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.255727 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.255786 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.255804 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 04:00:35.256768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.256089 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.256949 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257024 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4l9tj\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257093 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257341 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257456 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7tnpk\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257467 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257404 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 04:00:35.257687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257644 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k8j4f\"" Jul 09 04:00:35.258105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257803 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c"] Jul 09 04:00:35.258105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.257859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.258778 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.258599 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 04:00:35.258778 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.258676 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.258923 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.258881 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.259147 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.259130 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 04:00:35.259262 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.259242 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zdkcp\"" Jul 09 04:00:35.259810 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.259793 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8"] Jul 09 04:00:35.259923 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.259909 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.260718 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.260691 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.260801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.260697 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.261303 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.260944 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 04:00:35.261303 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.260990 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zmhrz\"" Jul 09 04:00:35.261978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.261960 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd"] Jul 09 04:00:35.262070 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.262061 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.263645 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.263627 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wc4zl"] Jul 09 04:00:35.263738 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.263728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.265256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.265237 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 04:00:35.266100 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.266081 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-jh6p7"] Jul 09 04:00:35.267245 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.267224 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.267337 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.267297 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.268418 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.268397 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.268816 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.268797 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 04:00:35.269383 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.269282 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 04:00:35.270424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.269594 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.270424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.269633 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xdjrw\"" Jul 09 04:00:35.274954 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.272301 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.274954 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.272490 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 04:00:35.274954 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.273682 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2r5p8\"" Jul 09 04:00:35.274954 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.273685 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vzksk\"" Jul 09 04:00:35.275796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.275775 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 04:00:35.277614 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.277246 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 04:00:35.278080 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.278060 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.278788 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.278768 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 04:00:35.279193 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.279167 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.280129 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.280109 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59497797c7-88275"] Jul 09 04:00:35.280226 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.280146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c"] Jul 09 04:00:35.280226 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.280217 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.280683 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.280662 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw"] Jul 09 04:00:35.281622 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.281601 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 04:00:35.282014 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.281994 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m"] Jul 09 04:00:35.282579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.282169 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 04:00:35.282579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.282431 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kqkcm\"" Jul 09 04:00:35.282579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.282472 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ggd2l"] Jul 09 04:00:35.284207 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.284189 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 04:00:35.284590 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.284572 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5s6xd"] Jul 09 04:00:35.285205 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.285141 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.285498 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.285474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 04:00:35.285498 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.285490 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.285630 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.285482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.286304 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.286283 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 04:00:35.286583 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.286565 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5bgcw\"" Jul 09 04:00:35.287307 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.287287 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm"] Jul 09 04:00:35.288704 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.288195 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pc8bq\"" Jul 09 04:00:35.288704 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.288238 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.288704 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.288469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.288941 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.288891 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.289064 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.289046 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.289956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd"] Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.290028 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-jh6p7"] Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.290054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.290192 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.290861 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 04:00:35.290997 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.290868 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 04:00:35.291784 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.291762 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2b4nw\"" Jul 09 04:00:35.294280 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.293455 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 04:00:35.294280 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.294059 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 04:00:35.294280 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.294122 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gq4xh\"" Jul 09 04:00:35.294750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.294410 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 04:00:35.294750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.294525 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggd2l"] Jul 09 04:00:35.294750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.294590 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5s6xd"] Jul 09 04:00:35.295391 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.295308 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wc4zl"] Jul 09 04:00:35.296698 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.296682 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm"] Jul 09 04:00:35.298442 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.298421 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8"] Jul 09 04:00:35.301830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.301805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms"] Jul 09 04:00:35.364095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ba1740-99fb-47a0-9c83-0689c076cc34-tmp-dir\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.364192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d09134d4-951d-4b4a-b251-a450d3176fad-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.364192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14d79e88-ae22-4f3c-8414-fbaca11408b2-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.364192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6gc\" (UniqueName: \"kubernetes.io/projected/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-kube-api-access-dq6gc\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.364192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f47ad52f-c4b5-4416-b63b-3e2ae739712a-cert\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364205 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xsj\" (UniqueName: \"kubernetes.io/projected/f47ad52f-c4b5-4416-b63b-3e2ae739712a-kube-api-access-46xsj\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-config\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364340 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlcc\" (UniqueName: \"kubernetes.io/projected/df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9-kube-api-access-hrlcc\") pod \"volume-data-source-validator-5d456fdbcd-fzjms\" (UID: \"df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xsxc\" (UniqueName: \"kubernetes.io/projected/d09134d4-951d-4b4a-b251-a450d3176fad-kube-api-access-4xsxc\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94ba1740-99fb-47a0-9c83-0689c076cc34-config-volume\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwxx\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.364626 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdn8j\" (UniqueName: \"kubernetes.io/projected/91335399-c3f6-414e-9086-0ce6b4f5061c-kube-api-access-mdn8j\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419714ad-2c57-4966-8ed2-173135fac77d-serving-cert\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364680 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h47b\" (UniqueName: \"kubernetes.io/projected/94ba1740-99fb-47a0-9c83-0689c076cc34-kube-api-access-6h47b\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-tmp\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4wt\" (UniqueName: \"kubernetes.io/projected/f89e374d-dac0-4a80-9412-f0e7a94d8a82-kube-api-access-8f4wt\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364772 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbkg\" (UniqueName: \"kubernetes.io/projected/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-kube-api-access-qjbkg\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfbx\" (UniqueName: \"kubernetes.io/projected/14d79e88-ae22-4f3c-8414-fbaca11408b2-kube-api-access-vdfbx\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14d79e88-ae22-4f3c-8414-fbaca11408b2-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364853 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-snapshots\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.364988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.364954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-serving-cert\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-trusted-ca\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rd5\" (UniqueName: \"kubernetes.io/projected/50e8d16f-b13e-4d9c-a20d-525c2c50165a-kube-api-access-h8rd5\") pod \"network-check-source-989f8b8dc-vmhmm\" (UID: \"50e8d16f-b13e-4d9c-a20d-525c2c50165a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365041 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365063 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-stats-auth\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwxn\" (UniqueName: \"kubernetes.io/projected/419714ad-2c57-4966-8ed2-173135fac77d-kube-api-access-czwxn\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e878bddb-eee6-4794-90be-2db53f9ad7cc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-default-certificate\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d09134d4-951d-4b4a-b251-a450d3176fad-config\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.365587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.365239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.466284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466223 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14d79e88-ae22-4f3c-8414-fbaca11408b2-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.466284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.466284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-snapshots\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-serving-cert\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.466403 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.466423 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-trusted-ca\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rd5\" (UniqueName: \"kubernetes.io/projected/50e8d16f-b13e-4d9c-a20d-525c2c50165a-kube-api-access-h8rd5\") pod \"network-check-source-989f8b8dc-vmhmm\" (UID: \"50e8d16f-b13e-4d9c-a20d-525c2c50165a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" Jul 09 04:00:35.466541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-stats-auth\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwxn\" (UniqueName: \"kubernetes.io/projected/419714ad-2c57-4966-8ed2-173135fac77d-kube-api-access-czwxn\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e878bddb-eee6-4794-90be-2db53f9ad7cc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-default-certificate\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d09134d4-951d-4b4a-b251-a450d3176fad-config\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ba1740-99fb-47a0-9c83-0689c076cc34-tmp-dir\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.466838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d09134d4-951d-4b4a-b251-a450d3176fad-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14d79e88-ae22-4f3c-8414-fbaca11408b2-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6gc\" (UniqueName: \"kubernetes.io/projected/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-kube-api-access-dq6gc\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f47ad52f-c4b5-4416-b63b-3e2ae739712a-cert\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xsj\" (UniqueName: \"kubernetes.io/projected/f47ad52f-c4b5-4416-b63b-3e2ae739712a-kube-api-access-46xsj\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-config\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlcc\" (UniqueName: \"kubernetes.io/projected/df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9-kube-api-access-hrlcc\") pod \"volume-data-source-validator-5d456fdbcd-fzjms\" (UID: \"df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.466996 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-snapshots\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xsxc\" (UniqueName: \"kubernetes.io/projected/d09134d4-951d-4b4a-b251-a450d3176fad-kube-api-access-4xsxc\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94ba1740-99fb-47a0-9c83-0689c076cc34-config-volume\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.467302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwxx\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdn8j\" (UniqueName: \"kubernetes.io/projected/91335399-c3f6-414e-9086-0ce6b4f5061c-kube-api-access-mdn8j\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.467329 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.967308092 +0000 UTC m=+35.571266824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467372 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-trusted-ca\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419714ad-2c57-4966-8ed2-173135fac77d-serving-cert\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467485 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h47b\" (UniqueName: \"kubernetes.io/projected/94ba1740-99fb-47a0-9c83-0689c076cc34-kube-api-access-6h47b\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-tmp\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4wt\" (UniqueName: \"kubernetes.io/projected/f89e374d-dac0-4a80-9412-f0e7a94d8a82-kube-api-access-8f4wt\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbkg\" (UniqueName: \"kubernetes.io/projected/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-kube-api-access-qjbkg\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfbx\" (UniqueName: \"kubernetes.io/projected/14d79e88-ae22-4f3c-8414-fbaca11408b2-kube-api-access-vdfbx\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.467638 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ba1740-99fb-47a0-9c83-0689c076cc34-tmp-dir\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.467714 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:35.467963 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.467764 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.967749147 +0000 UTC m=+35.571707877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:35.468591 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.467816 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:35.468591 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.467843 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.967834816 +0000 UTC m=+35.571793535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:35.468591 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.468173 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.468591 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.468413 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e878bddb-eee6-4794-90be-2db53f9ad7cc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.468884 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.469535 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.469628 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.469683 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.969668049 +0000 UTC m=+35.573626771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.470167 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94ba1740-99fb-47a0-9c83-0689c076cc34-config-volume\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.470466 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.470473 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-tmp\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.470515 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.970499432 +0000 UTC m=+35.574458155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.470610 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.470842 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.471100 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.471179 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.971159796 +0000 UTC m=+35.575118541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.471267 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:35.97124034 +0000 UTC m=+35.575199079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.471397 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d09134d4-951d-4b4a-b251-a450d3176fad-config\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.471862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.471557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.472693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.471917 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d09134d4-951d-4b4a-b251-a450d3176fad-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.472693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.471944 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14d79e88-ae22-4f3c-8414-fbaca11408b2-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.472693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.472052 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419714ad-2c57-4966-8ed2-173135fac77d-serving-cert\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.472693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.472443 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-serving-cert\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.473028 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.472979 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-stats-auth\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.473557 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.473429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/419714ad-2c57-4966-8ed2-173135fac77d-config\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.473662 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.473643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.473721 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.473675 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14d79e88-ae22-4f3c-8414-fbaca11408b2-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.473984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.473966 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.475167 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.475138 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f47ad52f-c4b5-4416-b63b-3e2ae739712a-cert\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.475543 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.475519 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-default-certificate\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.476817 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.476792 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.478974 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.478951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rd5\" (UniqueName: \"kubernetes.io/projected/50e8d16f-b13e-4d9c-a20d-525c2c50165a-kube-api-access-h8rd5\") pod \"network-check-source-989f8b8dc-vmhmm\" (UID: \"50e8d16f-b13e-4d9c-a20d-525c2c50165a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" Jul 09 04:00:35.479442 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.479422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdn8j\" (UniqueName: \"kubernetes.io/projected/91335399-c3f6-414e-9086-0ce6b4f5061c-kube-api-access-mdn8j\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.479603 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.479557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwxn\" (UniqueName: \"kubernetes.io/projected/419714ad-2c57-4966-8ed2-173135fac77d-kube-api-access-czwxn\") pod \"console-operator-68b757fb67-jh6p7\" (UID: \"419714ad-2c57-4966-8ed2-173135fac77d\") " pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.480094 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.480061 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwxx\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.480273 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.480253 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h47b\" (UniqueName: \"kubernetes.io/projected/94ba1740-99fb-47a0-9c83-0689c076cc34-kube-api-access-6h47b\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.480337 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.480253 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfbx\" (UniqueName: \"kubernetes.io/projected/14d79e88-ae22-4f3c-8414-fbaca11408b2-kube-api-access-vdfbx\") pod \"kube-storage-version-migrator-operator-846bf668f7-k7n5c\" (UID: \"14d79e88-ae22-4f3c-8414-fbaca11408b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.480421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.480395 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xsj\" (UniqueName: \"kubernetes.io/projected/f47ad52f-c4b5-4416-b63b-3e2ae739712a-kube-api-access-46xsj\") pod \"ingress-canary-ggd2l\" (UID: \"f47ad52f-c4b5-4416-b63b-3e2ae739712a\") " pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.481462 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.481444 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4wt\" (UniqueName: \"kubernetes.io/projected/f89e374d-dac0-4a80-9412-f0e7a94d8a82-kube-api-access-8f4wt\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.481557 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.481476 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xsxc\" (UniqueName: \"kubernetes.io/projected/d09134d4-951d-4b4a-b251-a450d3176fad-kube-api-access-4xsxc\") pod \"service-ca-operator-5d46d9ffd4-5fjkw\" (UID: \"d09134d4-951d-4b4a-b251-a450d3176fad\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.485946 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.485922 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbkg\" (UniqueName: \"kubernetes.io/projected/ce0cf1d9-9b4a-4dae-9333-0973ae5928d3-kube-api-access-qjbkg\") pod \"insights-operator-5c847dcfc4-wc4zl\" (UID: \"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3\") " pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.486868 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.486850 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlcc\" (UniqueName: \"kubernetes.io/projected/df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9-kube-api-access-hrlcc\") pod \"volume-data-source-validator-5d456fdbcd-fzjms\" (UID: \"df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" Jul 09 04:00:35.487184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.487166 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6gc\" (UniqueName: \"kubernetes.io/projected/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-kube-api-access-dq6gc\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.581738 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.581711 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" Jul 09 04:00:35.593320 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.593298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" Jul 09 04:00:35.611522 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.611493 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" Jul 09 04:00:35.632725 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.632696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" Jul 09 04:00:35.641019 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.640996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:35.649020 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.648763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggd2l" Jul 09 04:00:35.661669 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.661395 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" Jul 09 04:00:35.841602 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.841471 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw"] Jul 09 04:00:35.843176 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.842335 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms"] Jul 09 04:00:35.847258 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.844641 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09134d4_951d_4b4a_b251_a450d3176fad.slice/crio-f00982c2e3d4774b859a0b7d3c7585cdf4767c1c177af34b9a49204695c62665 WatchSource:0}: Error finding container f00982c2e3d4774b859a0b7d3c7585cdf4767c1c177af34b9a49204695c62665: Status 404 returned error can't find the container with id f00982c2e3d4774b859a0b7d3c7585cdf4767c1c177af34b9a49204695c62665 Jul 09 04:00:35.850008 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.849919 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0f4a8d_f57a_4be0_b7f6_30ed0d655ec9.slice/crio-7bd64be5c016fa4046cb6525f3aef0ee651ccbc522c66770a76d7eef1583ce9a WatchSource:0}: Error finding container 7bd64be5c016fa4046cb6525f3aef0ee651ccbc522c66770a76d7eef1583ce9a: Status 404 returned error can't find the container with id 7bd64be5c016fa4046cb6525f3aef0ee651ccbc522c66770a76d7eef1583ce9a Jul 09 04:00:35.873888 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.873739 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wc4zl"] Jul 09 04:00:35.875542 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.875486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c"] Jul 09 04:00:35.876445 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.876326 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0cf1d9_9b4a_4dae_9333_0973ae5928d3.slice/crio-0217324d6b184f5eb01567043a32a59211be619fb0265ba5dc314b2062c1bc5d WatchSource:0}: Error finding container 0217324d6b184f5eb01567043a32a59211be619fb0265ba5dc314b2062c1bc5d: Status 404 returned error can't find the container with id 0217324d6b184f5eb01567043a32a59211be619fb0265ba5dc314b2062c1bc5d Jul 09 04:00:35.880085 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.880049 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d79e88_ae22_4f3c_8414_fbaca11408b2.slice/crio-6c444ab78dfcb30106e9692a2e6e206afe6c4be1b3b0ce24497243f79fd95d6a WatchSource:0}: Error finding container 6c444ab78dfcb30106e9692a2e6e206afe6c4be1b3b0ce24497243f79fd95d6a: Status 404 returned error can't find the container with id 6c444ab78dfcb30106e9692a2e6e206afe6c4be1b3b0ce24497243f79fd95d6a Jul 09 04:00:35.891030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.890986 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-jh6p7"] Jul 09 04:00:35.894078 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.894053 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419714ad_2c57_4966_8ed2_173135fac77d.slice/crio-c616f3db8617b1cc8d2c5b46a9ef5baf90c5b7468c2ec5223c8217a65058a759 WatchSource:0}: Error finding container c616f3db8617b1cc8d2c5b46a9ef5baf90c5b7468c2ec5223c8217a65058a759: Status 404 returned error can't find the container with id c616f3db8617b1cc8d2c5b46a9ef5baf90c5b7468c2ec5223c8217a65058a759 Jul 09 04:00:35.897588 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.897535 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggd2l"] Jul 09 04:00:35.901880 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.901854 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47ad52f_c4b5_4416_b63b_3e2ae739712a.slice/crio-924452d872171e4f5416bb356f3fa06e9b55cd55e9809806f3824b583661866d WatchSource:0}: Error finding container 924452d872171e4f5416bb356f3fa06e9b55cd55e9809806f3824b583661866d: Status 404 returned error can't find the container with id 924452d872171e4f5416bb356f3fa06e9b55cd55e9809806f3824b583661866d Jul 09 04:00:35.908610 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.908558 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm"] Jul 09 04:00:35.910941 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:00:35.910919 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e8d16f_b13e_4d9c_a20d_525c2c50165a.slice/crio-107adcf5bc8be26e4e7da080f937e2f8d10739ab65d00a1ef340553e547466c7 WatchSource:0}: Error finding container 107adcf5bc8be26e4e7da080f937e2f8d10739ab65d00a1ef340553e547466c7: Status 404 returned error can't find the container with id 107adcf5bc8be26e4e7da080f937e2f8d10739ab65d00a1ef340553e547466c7 Jul 09 04:00:35.975644 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.975644 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975722 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975785 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.97576543 +0000 UTC m=+36.579724152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975802 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975807 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975821 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.97580197 +0000 UTC m=+36.579760691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:35.975847 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975847 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.975836578 +0000 UTC m=+36.579795300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.975870 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.975861157 +0000 UTC m=+36.579819876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.975996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:35.976029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976050 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976064 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976107 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.976090927 +0000 UTC m=+36.580049665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976127 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:35.976155 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976132 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:35.976586 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976174 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.97616259 +0000 UTC m=+36.580121310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:35.976586 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:35.976192 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:36.976182818 +0000 UTC m=+36.580141543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:36.070532 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.070507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" event={"ID":"14d79e88-ae22-4f3c-8414-fbaca11408b2","Type":"ContainerStarted","Data":"6c444ab78dfcb30106e9692a2e6e206afe6c4be1b3b0ce24497243f79fd95d6a"} Jul 09 04:00:36.071839 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.071803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" event={"ID":"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3","Type":"ContainerStarted","Data":"0217324d6b184f5eb01567043a32a59211be619fb0265ba5dc314b2062c1bc5d"} Jul 09 04:00:36.073156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.073136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" event={"ID":"df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9","Type":"ContainerStarted","Data":"7bd64be5c016fa4046cb6525f3aef0ee651ccbc522c66770a76d7eef1583ce9a"} Jul 09 04:00:36.074425 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.074406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" event={"ID":"50e8d16f-b13e-4d9c-a20d-525c2c50165a","Type":"ContainerStarted","Data":"107adcf5bc8be26e4e7da080f937e2f8d10739ab65d00a1ef340553e547466c7"} Jul 09 04:00:36.075722 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.075702 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggd2l" event={"ID":"f47ad52f-c4b5-4416-b63b-3e2ae739712a","Type":"ContainerStarted","Data":"924452d872171e4f5416bb356f3fa06e9b55cd55e9809806f3824b583661866d"} Jul 09 04:00:36.076929 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.076907 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" event={"ID":"419714ad-2c57-4966-8ed2-173135fac77d","Type":"ContainerStarted","Data":"c616f3db8617b1cc8d2c5b46a9ef5baf90c5b7468c2ec5223c8217a65058a759"} Jul 09 04:00:36.078019 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.077997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" event={"ID":"d09134d4-951d-4b4a-b251-a450d3176fad","Type":"ContainerStarted","Data":"f00982c2e3d4774b859a0b7d3c7585cdf4767c1c177af34b9a49204695c62665"} Jul 09 04:00:36.917474 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.917226 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:00:36.917907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.917264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:00:36.917907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.917329 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:00:36.925208 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.922461 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vbqhw\"" Jul 09 04:00:36.925208 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.922809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 04:00:36.925208 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.922989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 04:00:36.925208 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.923179 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqsq9\"" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:36.986648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.986792 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.986773791 +0000 UTC m=+38.590732512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987202 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987260 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.98723771 +0000 UTC m=+38.591196445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987323 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987371 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.987343454 +0000 UTC m=+38.591302189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987432 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987443 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987474 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.987464118 +0000 UTC m=+38.591422859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:36.987822 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987549 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:36.988768 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987580 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.987570196 +0000 UTC m=+38.591528915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:36.988768 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987647 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:36.988768 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987687 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.987677123 +0000 UTC m=+38.591635845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:36.988768 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987740 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:36.988768 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:36.987768 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:38.98775886 +0000 UTC m=+38.591717594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.008901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.008998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.009033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009064 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009085 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009145 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009123626 +0000 UTC m=+42.613082348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009146 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009180 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009170297 +0000 UTC m=+42.613129015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.009066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:39.009216 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009193 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.009242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009263 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.009281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009299 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009283285 +0000 UTC m=+42.613242004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009333 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:39.009366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009380 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009369052 +0000 UTC m=+42.613327775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009464 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009447822 +0000 UTC m=+42.613406542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009545 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009534842 +0000 UTC m=+42.613493562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009583 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:39.010229 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:39.009619 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:00:43.009608001 +0000 UTC m=+42.613566723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:43.045843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.045794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.045863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.045935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.045964 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046032 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046014194 +0000 UTC m=+50.649972912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.045971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.046082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046030 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.046125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046126 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:43.046199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046049 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046247 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046230736 +0000 UTC m=+50.650189469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046055 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046190 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046264 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046255159 +0000 UTC m=+50.650213886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046323 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046312206 +0000 UTC m=+50.650270952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:43.046421 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046327 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:43.047098 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046343 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046333021 +0000 UTC m=+50.650291742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:43.047098 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046389 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.046378994 +0000 UTC m=+50.650337721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:43.047098 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:43.046407 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:51.04639841 +0000 UTC m=+50.650357128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:46.109798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.109763 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" event={"ID":"d09134d4-951d-4b4a-b251-a450d3176fad","Type":"ContainerStarted","Data":"32f7d29c757e49697965989a9b5098922e527827b9e134d589035b8b3729858d"} Jul 09 04:00:46.111873 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.111847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" event={"ID":"14d79e88-ae22-4f3c-8414-fbaca11408b2","Type":"ContainerStarted","Data":"fd167fc4b85bf0cc91fee106f11cff576f233e968b4f05611a4ff88a3498ce3e"} Jul 09 04:00:46.115150 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.115126 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="f03d83678d7f00292281d6990135d80532872f70a011a3c83427949817f561e0" exitCode=0 Jul 09 04:00:46.115258 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.115204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"f03d83678d7f00292281d6990135d80532872f70a011a3c83427949817f561e0"} Jul 09 04:00:46.116811 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.116786 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" event={"ID":"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3","Type":"ContainerStarted","Data":"395476b55c72a5f292e20e4dee348c453e5d399192164387e280872e7a70227f"} Jul 09 04:00:46.118335 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.118315 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" event={"ID":"df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9","Type":"ContainerStarted","Data":"4fcee1ab4b9495153e5f5b274a0c5a7df7143526fa9613ec276f7363c4e8b1d6"} Jul 09 04:00:46.123661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.123638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" event={"ID":"50e8d16f-b13e-4d9c-a20d-525c2c50165a","Type":"ContainerStarted","Data":"e52a23e0bb7a3e361cf68282cbe15d97d6e39086677baa01daccf2b84359aa77"} Jul 09 04:00:46.125842 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.125794 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" podStartSLOduration=33.600515584 podStartE2EDuration="43.12577721s" totalImagesPullingTime="9.525261626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.124283038 +0000 UTC m=+45.728241782" watchObservedRunningTime="2026-07-09 04:00:46.12577721 +0000 UTC m=+45.729735954" Jul 09 04:00:46.126433 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.126342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggd2l" event={"ID":"f47ad52f-c4b5-4416-b63b-3e2ae739712a","Type":"ContainerStarted","Data":"f3b597a5e2c5af9cc2cb1f3fa8b298d7dc0159dc33d1d8f4a7e522dfc3da07fe"} Jul 09 04:00:46.128493 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.128474 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/0.log" Jul 09 04:00:46.128587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.128512 2570 generic.go:356] "Generic (PLEG): container finished" podID="419714ad-2c57-4966-8ed2-173135fac77d" containerID="ac857bce3f2f781fab7d1c5117b5746d208860ac70359ba428ce6114558605ca" exitCode=255 Jul 09 04:00:46.128587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.128541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" event={"ID":"419714ad-2c57-4966-8ed2-173135fac77d","Type":"ContainerDied","Data":"ac857bce3f2f781fab7d1c5117b5746d208860ac70359ba428ce6114558605ca"} Jul 09 04:00:46.128774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.128757 2570 scope.go:117] "RemoveContainer" containerID="ac857bce3f2f781fab7d1c5117b5746d208860ac70359ba428ce6114558605ca" Jul 09 04:00:46.138714 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.138677 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fzjms" podStartSLOduration=33.230873767 podStartE2EDuration="43.138665063s" totalImagesPullingTime="9.907791296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.137610698 +0000 UTC m=+45.741569439" watchObservedRunningTime="2026-07-09 04:00:46.138665063 +0000 UTC m=+45.742623807" Jul 09 04:00:46.154072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.154031 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" podStartSLOduration=33.663768275 podStartE2EDuration="43.154014587s" totalImagesPullingTime="9.490246312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.152891196 +0000 UTC m=+45.756849935" watchObservedRunningTime="2026-07-09 04:00:46.154014587 +0000 UTC m=+45.757973329" Jul 09 04:00:46.169080 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.169039 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" podStartSLOduration=33.673814425 podStartE2EDuration="43.169023213s" totalImagesPullingTime="9.495208788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.168893647 +0000 UTC m=+45.772852389" watchObservedRunningTime="2026-07-09 04:00:46.169023213 +0000 UTC m=+45.772981954" Jul 09 04:00:46.255529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.255376 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ggd2l" podStartSLOduration=1.784609256 podStartE2EDuration="11.255336138s" totalImagesPullingTime="9.470726882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.253477498 +0000 UTC m=+45.857436240" watchObservedRunningTime="2026-07-09 04:00:46.255336138 +0000 UTC m=+45.859294880" Jul 09 04:00:46.255529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:46.255483 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-vmhmm" podStartSLOduration=33.79374422 podStartE2EDuration="43.255476054s" totalImagesPullingTime="9.461731834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:46.238732631 +0000 UTC m=+45.842691372" watchObservedRunningTime="2026-07-09 04:00:46.255476054 +0000 UTC m=+45.859434794" Jul 09 04:00:47.137510 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.137472 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9787e2f-3fde-4bb3-8d4d-a058c3b46757" containerID="e0e66a56746a8457e86258d45a4417e958dfac27a399743371711c86fa3e3243" exitCode=0 Jul 09 04:00:47.137987 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.137565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerDied","Data":"e0e66a56746a8457e86258d45a4417e958dfac27a399743371711c86fa3e3243"} Jul 09 04:00:47.139214 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139197 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:00:47.139614 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139599 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/0.log" Jul 09 04:00:47.139710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139634 2570 generic.go:356] "Generic (PLEG): container finished" podID="419714ad-2c57-4966-8ed2-173135fac77d" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" exitCode=255 Jul 09 04:00:47.139768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" event={"ID":"419714ad-2c57-4966-8ed2-173135fac77d","Type":"ContainerDied","Data":"93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180"} Jul 09 04:00:47.139808 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139796 2570 scope.go:117] "RemoveContainer" containerID="ac857bce3f2f781fab7d1c5117b5746d208860ac70359ba428ce6114558605ca" Jul 09 04:00:47.140008 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:47.139986 2570 scope.go:117] "RemoveContainer" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" Jul 09 04:00:47.140184 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:47.140164 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-jh6p7_openshift-console-operator(419714ad-2c57-4966-8ed2-173135fac77d)\"" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" podUID="419714ad-2c57-4966-8ed2-173135fac77d" Jul 09 04:00:48.147245 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.147210 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" event={"ID":"e9787e2f-3fde-4bb3-8d4d-a058c3b46757","Type":"ContainerStarted","Data":"9b439b53ae187fa610bb03f3f6b3e8f0959a29c1c761bd93226bf0263d990fda"} Jul 09 04:00:48.148836 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.148814 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:00:48.149075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.149061 2570 scope.go:117] "RemoveContainer" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" Jul 09 04:00:48.149208 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:48.149189 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-jh6p7_openshift-console-operator(419714ad-2c57-4966-8ed2-173135fac77d)\"" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" podUID="419714ad-2c57-4966-8ed2-173135fac77d" Jul 09 04:00:48.173551 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.173514 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6p5qz" podStartSLOduration=8.422721863 podStartE2EDuration="47.173499565s" totalImagesPullingTime="38.750777702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:48.172736977 +0000 UTC m=+47.776695719" watchObservedRunningTime="2026-07-09 04:00:48.173499565 +0000 UTC m=+47.777458307" Jul 09 04:00:48.622943 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.622912 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pppkm"] Jul 09 04:00:48.640864 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.640832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pppkm"] Jul 09 04:00:48.640999 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.640910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.642828 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.642809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwpz9\"" Jul 09 04:00:48.642962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.642828 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 04:00:48.642962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.642828 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 04:00:48.693157 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.693135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-data-volume\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.693292 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.693185 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.693292 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.693231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.693434 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.693312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gq48\" (UniqueName: \"kubernetes.io/projected/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-api-access-8gq48\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.693489 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.693445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-crio-socket\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794209 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794314 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gq48\" (UniqueName: \"kubernetes.io/projected/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-api-access-8gq48\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794397 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:48.794310 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:48.794472 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:48.794402 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:49.294380783 +0000 UTC m=+48.898339505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : configmap "kube-rbac-proxy" not found Jul 09 04:00:48.794544 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-crio-socket\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794606 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-data-volume\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794640 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-crio-socket\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.794758 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:48.794727 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:48.794809 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:48.794779 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:49.294764798 +0000 UTC m=+48.898723518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:48.794876 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.794860 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-data-volume\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:48.802905 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:48.802878 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gq48\" (UniqueName: \"kubernetes.io/projected/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-api-access-8gq48\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:49.299661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:49.299629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:49.299661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:49.299663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:49.300051 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:49.299736 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:49.300051 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:49.299789 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:50.299774323 +0000 UTC m=+49.903733042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : configmap "kube-rbac-proxy" not found Jul 09 04:00:49.300051 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:49.299814 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:49.300051 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:49.299905 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:50.299887075 +0000 UTC m=+49.903845794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:50.310772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:50.310735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:50.310772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:50.310776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:50.311243 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:50.310871 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:50.311243 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:50.310944 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:52.310926668 +0000 UTC m=+51.914885409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:50.311243 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:50.310961 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:50.311243 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:50.311033 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:52.31101352 +0000 UTC m=+51.914972255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : configmap "kube-rbac-proxy" not found Jul 09 04:00:50.804394 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:50.804368 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-78sl6_d4bae520-6f0d-47e6-b75c-950c05499b3c/dns-node-resolver/0.log" Jul 09 04:00:51.119231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119295 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119366 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls podName:91335399-c3f6-414e-9086-0ce6b4f5061c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119333209 +0000 UTC m=+66.723291927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wgp2m" (UID: "91335399-c3f6-414e-9086-0ce6b4f5061c") : secret "samples-operator-tls" not found Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119376 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119392 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cd586689-zfs5d: secret "image-registry-tls" not found Jul 09 04:00:51.119410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119419 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119436 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls podName:be11c46e-b7a1-4194-b71f-6c59140f8c1d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119420353 +0000 UTC m=+66.723379089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls") pod "image-registry-79cd586689-zfs5d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d") : secret "image-registry-tls" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119439 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119460 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls podName:a0e1b35d-5a64-4eaa-a85c-9899af48c4a6 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119451057 +0000 UTC m=+66.723409798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7d4qd" (UID: "a0e1b35d-5a64-4eaa-a85c-9899af48c4a6") : secret "cluster-monitoring-operator-tls" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119487 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert podName:e878bddb-eee6-4794-90be-2db53f9ad7cc nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119472838 +0000 UTC m=+66.723431580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pkbg8" (UID: "e878bddb-eee6-4794-90be-2db53f9ad7cc") : secret "networking-console-plugin-cert" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119493 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119542 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119530858 +0000 UTC m=+66.723489589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : secret "router-metrics-certs-default" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:51.119587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119598 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle podName:f89e374d-dac0-4a80-9412-f0e7a94d8a82 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.11958792 +0000 UTC m=+66.723546668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle") pod "router-default-59497797c7-88275" (UID: "f89e374d-dac0-4a80-9412-f0e7a94d8a82") : configmap references non-existent config key: service-ca.crt Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119626 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:51.119699 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:51.119647 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls podName:94ba1740-99fb-47a0-9c83-0689c076cc34 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:07.119639978 +0000 UTC m=+66.723598697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls") pod "dns-default-5s6xd" (UID: "94ba1740-99fb-47a0-9c83-0689c076cc34") : secret "dns-default-metrics-tls" not found Jul 09 04:00:52.004213 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:52.004188 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-djx94_81c5e01e-0e76-493e-9a70-9125d05cd963/node-ca/0.log" Jul 09 04:00:52.329918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:52.329895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:52.330056 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:52.329925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:52.330105 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:52.330053 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:52.330144 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:52.330137 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:00:56.330106938 +0000 UTC m=+55.934065656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:52.330430 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:52.330403 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:53.004044 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:53.004015 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggd2l_f47ad52f-c4b5-4416-b63b-3e2ae739712a/serve-healthcheck-canary/0.log" Jul 09 04:00:53.604777 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:53.604738 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-k7n5c_14d79e88-ae22-4f3c-8414-fbaca11408b2/kube-storage-version-migrator-operator/0.log" Jul 09 04:00:55.641368 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:55.641316 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:55.642019 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:55.641396 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:00:55.642019 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:55.641843 2570 scope.go:117] "RemoveContainer" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" Jul 09 04:00:55.642141 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:55.642039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-jh6p7_openshift-console-operator(419714ad-2c57-4966-8ed2-173135fac77d)\"" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" podUID="419714ad-2c57-4966-8ed2-173135fac77d" Jul 09 04:00:56.174183 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:56.174152 2570 scope.go:117] "RemoveContainer" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" Jul 09 04:00:56.174380 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:56.174330 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-jh6p7_openshift-console-operator(419714ad-2c57-4966-8ed2-173135fac77d)\"" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" podUID="419714ad-2c57-4966-8ed2-173135fac77d" Jul 09 04:00:56.362092 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:00:56.362067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:00:56.362234 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:56.362208 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:56.362304 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:00:56.362272 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls podName:b2c5f642-6ed5-442b-8e2c-e74ee36237bc nodeName:}" failed. No retries permitted until 2026-07-09 04:01:04.362253051 +0000 UTC m=+63.966211770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pppkm" (UID: "b2c5f642-6ed5-442b-8e2c-e74ee36237bc") : secret "insights-runtime-extractor-tls" not found Jul 09 04:01:00.064265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:00.064240 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqqng" Jul 09 04:01:04.423733 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:04.423696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:01:04.425930 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:04.425911 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2c5f642-6ed5-442b-8e2c-e74ee36237bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pppkm\" (UID: \"b2c5f642-6ed5-442b-8e2c-e74ee36237bc\") " pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:01:04.551453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:04.551429 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwpz9\"" Jul 09 04:01:04.560059 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:04.560039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pppkm" Jul 09 04:01:04.675009 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:04.674947 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pppkm"] Jul 09 04:01:04.678226 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:04.678197 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c5f642_6ed5_442b_8e2c_e74ee36237bc.slice/crio-e75a31c6175523346123a5e9fe6ebdfd7bfa0b948c65596e89701cf5e298990e WatchSource:0}: Error finding container e75a31c6175523346123a5e9fe6ebdfd7bfa0b948c65596e89701cf5e298990e: Status 404 returned error can't find the container with id e75a31c6175523346123a5e9fe6ebdfd7bfa0b948c65596e89701cf5e298990e Jul 09 04:01:05.203528 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:05.203443 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pppkm" event={"ID":"b2c5f642-6ed5-442b-8e2c-e74ee36237bc","Type":"ContainerStarted","Data":"e75a31c6175523346123a5e9fe6ebdfd7bfa0b948c65596e89701cf5e298990e"} Jul 09 04:01:06.643894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.643859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:01:06.643894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.643900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:01:06.646423 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.646394 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 04:01:06.646769 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.646748 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 04:01:06.656751 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.656723 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd95163-b562-4bd6-9157-c69ebc28e76b-metrics-certs\") pod \"network-metrics-daemon-xp2dp\" (UID: \"2fd95163-b562-4bd6-9157-c69ebc28e76b\") " pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:01:06.656930 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.656910 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f4e347-8cf3-4ecc-b6c1-18e9c590e662-original-pull-secret\") pod \"global-pull-secret-syncer-54pxr\" (UID: \"21f4e347-8cf3-4ecc-b6c1-18e9c590e662\") " pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:01:06.745052 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.745014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:01:06.747772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.747721 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4vk\" (UniqueName: \"kubernetes.io/projected/7317d19d-eafb-4286-aec8-896345117a22-kube-api-access-rj4vk\") pod \"network-check-target-vfs25\" (UID: \"7317d19d-eafb-4286-aec8-896345117a22\") " pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:01:06.936028 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.935957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54pxr" Jul 09 04:01:06.949811 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.949786 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vbqhw\"" Jul 09 04:01:06.956881 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.956863 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqsq9\"" Jul 09 04:01:06.957899 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.957882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:01:06.965598 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:06.965578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp2dp" Jul 09 04:01:07.148450 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:01:07.148625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.148599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:07.149166 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.149115 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89e374d-dac0-4a80-9412-f0e7a94d8a82-service-ca-bundle\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:07.151559 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.151528 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e878bddb-eee6-4794-90be-2db53f9ad7cc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pkbg8\" (UID: \"e878bddb-eee6-4794-90be-2db53f9ad7cc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:01:07.151678 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.151573 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ba1740-99fb-47a0-9c83-0689c076cc34-metrics-tls\") pod \"dns-default-5s6xd\" (UID: \"94ba1740-99fb-47a0-9c83-0689c076cc34\") " pod="openshift-dns/dns-default-5s6xd" Jul 09 04:01:07.151951 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.151932 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0e1b35d-5a64-4eaa-a85c-9899af48c4a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7d4qd\" (UID: \"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:01:07.152034 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.151957 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"image-registry-79cd586689-zfs5d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:07.152087 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.152070 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91335399-c3f6-414e-9086-0ce6b4f5061c-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wgp2m\" (UID: \"91335399-c3f6-414e-9086-0ce6b4f5061c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:01:07.152128 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.152070 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89e374d-dac0-4a80-9412-f0e7a94d8a82-metrics-certs\") pod \"router-default-59497797c7-88275\" (UID: \"f89e374d-dac0-4a80-9412-f0e7a94d8a82\") " pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:07.160321 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.158480 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2b4nw\"" Jul 09 04:01:07.165926 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.165905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5s6xd" Jul 09 04:01:07.363359 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.363333 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4l9tj\"" Jul 09 04:01:07.370822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.370801 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k8j4f\"" Jul 09 04:01:07.371886 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.371860 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:07.379789 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.379768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:07.406880 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.406860 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zmhrz\"" Jul 09 04:01:07.415463 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.415445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" Jul 09 04:01:07.421511 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.421487 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xdjrw\"" Jul 09 04:01:07.427840 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.427820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vzksk\"" Jul 09 04:01:07.429206 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.429177 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" Jul 09 04:01:07.436888 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:07.436869 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" Jul 09 04:01:08.470421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.470291 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5s6xd"] Jul 09 04:01:08.472584 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.472562 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ba1740_99fb_47a0_9c83_0689c076cc34.slice/crio-13cd9c107fd1a4422668d2c46843a0d79cab4b1979160e4c1e3ec1d7fea3e195 WatchSource:0}: Error finding container 13cd9c107fd1a4422668d2c46843a0d79cab4b1979160e4c1e3ec1d7fea3e195: Status 404 returned error can't find the container with id 13cd9c107fd1a4422668d2c46843a0d79cab4b1979160e4c1e3ec1d7fea3e195 Jul 09 04:01:08.533443 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.533412 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:01:08.536496 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.536469 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe11c46e_b7a1_4194_b71f_6c59140f8c1d.slice/crio-a82b7d5cd60a0df1113d6ef2a35e894f6aeae72022ef3e7b4afc89a7a8e4a7e4 WatchSource:0}: Error finding container a82b7d5cd60a0df1113d6ef2a35e894f6aeae72022ef3e7b4afc89a7a8e4a7e4: Status 404 returned error can't find the container with id a82b7d5cd60a0df1113d6ef2a35e894f6aeae72022ef3e7b4afc89a7a8e4a7e4 Jul 09 04:01:08.693696 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.693661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfs25"] Jul 09 04:01:08.696674 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.696633 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7317d19d_eafb_4286_aec8_896345117a22.slice/crio-f598a876f93caf258c3b6264e7764e92c24f7ca74d7be582ae698fb992404df3 WatchSource:0}: Error finding container f598a876f93caf258c3b6264e7764e92c24f7ca74d7be582ae698fb992404df3: Status 404 returned error can't find the container with id f598a876f93caf258c3b6264e7764e92c24f7ca74d7be582ae698fb992404df3 Jul 09 04:01:08.697567 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.697538 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-54pxr"] Jul 09 04:01:08.698599 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.698435 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xp2dp"] Jul 09 04:01:08.701114 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.701090 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd95163_b562_4bd6_9157_c69ebc28e76b.slice/crio-82f5d308e9392f5f58f69c374bfd4a90acb27e5ac8812fc5b4f84441eb0b9051 WatchSource:0}: Error finding container 82f5d308e9392f5f58f69c374bfd4a90acb27e5ac8812fc5b4f84441eb0b9051: Status 404 returned error can't find the container with id 82f5d308e9392f5f58f69c374bfd4a90acb27e5ac8812fc5b4f84441eb0b9051 Jul 09 04:01:08.703421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.703398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59497797c7-88275"] Jul 09 04:01:08.708044 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.708018 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89e374d_dac0_4a80_9412_f0e7a94d8a82.slice/crio-fa677383c49fcd57e1b82706a9bac5c3b23649aff3753bbe9b102d52771c69d6 WatchSource:0}: Error finding container fa677383c49fcd57e1b82706a9bac5c3b23649aff3753bbe9b102d52771c69d6: Status 404 returned error can't find the container with id fa677383c49fcd57e1b82706a9bac5c3b23649aff3753bbe9b102d52771c69d6 Jul 09 04:01:08.754727 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.754646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8"] Jul 09 04:01:08.757924 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.757895 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode878bddb_eee6_4794_90be_2db53f9ad7cc.slice/crio-f654eb8e1f6dc6d992fcb33c4530d5b6e7e53f57681455163934313da5bf9a72 WatchSource:0}: Error finding container f654eb8e1f6dc6d992fcb33c4530d5b6e7e53f57681455163934313da5bf9a72: Status 404 returned error can't find the container with id f654eb8e1f6dc6d992fcb33c4530d5b6e7e53f57681455163934313da5bf9a72 Jul 09 04:01:08.758580 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.758550 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd"] Jul 09 04:01:08.759948 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:08.759818 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m"] Jul 09 04:01:08.762793 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:08.762765 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e1b35d_5a64_4eaa_a85c_9899af48c4a6.slice/crio-a749f3a9721411b1eeee60d75bde5fc9d40d8410411bfa1207d70cecbf4576c4 WatchSource:0}: Error finding container a749f3a9721411b1eeee60d75bde5fc9d40d8410411bfa1207d70cecbf4576c4: Status 404 returned error can't find the container with id a749f3a9721411b1eeee60d75bde5fc9d40d8410411bfa1207d70cecbf4576c4 Jul 09 04:01:09.219577 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.219520 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" event={"ID":"be11c46e-b7a1-4194-b71f-6c59140f8c1d","Type":"ContainerStarted","Data":"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6"} Jul 09 04:01:09.219718 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.219585 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:09.219718 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.219608 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" event={"ID":"be11c46e-b7a1-4194-b71f-6c59140f8c1d","Type":"ContainerStarted","Data":"a82b7d5cd60a0df1113d6ef2a35e894f6aeae72022ef3e7b4afc89a7a8e4a7e4"} Jul 09 04:01:09.220942 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.220916 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-54pxr" event={"ID":"21f4e347-8cf3-4ecc-b6c1-18e9c590e662","Type":"ContainerStarted","Data":"a2a4d91bb4edd4ea8f74b05280a9986526024d7d002313fa2b16be92d22af3d4"} Jul 09 04:01:09.222210 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.222189 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp2dp" event={"ID":"2fd95163-b562-4bd6-9157-c69ebc28e76b","Type":"ContainerStarted","Data":"82f5d308e9392f5f58f69c374bfd4a90acb27e5ac8812fc5b4f84441eb0b9051"} Jul 09 04:01:09.223804 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.223781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" event={"ID":"e878bddb-eee6-4794-90be-2db53f9ad7cc","Type":"ContainerStarted","Data":"f654eb8e1f6dc6d992fcb33c4530d5b6e7e53f57681455163934313da5bf9a72"} Jul 09 04:01:09.225922 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.225893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pppkm" event={"ID":"b2c5f642-6ed5-442b-8e2c-e74ee36237bc","Type":"ContainerStarted","Data":"cea5019d43b9f85a2b404901ba3beee5d904e4a6a9de68dceb0cf5100a1ce893"} Jul 09 04:01:09.226010 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.225942 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pppkm" event={"ID":"b2c5f642-6ed5-442b-8e2c-e74ee36237bc","Type":"ContainerStarted","Data":"6744ea2aff00ef2dfac067ca114a96451b8c2d11d61d7d61b72fb1cabc15c4d1"} Jul 09 04:01:09.227210 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.227179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" event={"ID":"91335399-c3f6-414e-9086-0ce6b4f5061c","Type":"ContainerStarted","Data":"20ef686a4a72c843dd73e0a4ec0379b77faf15b37ccecd97cf0eae70a527d46c"} Jul 09 04:01:09.228949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.228840 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfs25" event={"ID":"7317d19d-eafb-4286-aec8-896345117a22","Type":"ContainerStarted","Data":"8cdd83b2742bc19fdb7604814f3eb025fcc13a03ba8ed8ed4de0dfef4d38b405"} Jul 09 04:01:09.228949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.228877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfs25" event={"ID":"7317d19d-eafb-4286-aec8-896345117a22","Type":"ContainerStarted","Data":"f598a876f93caf258c3b6264e7764e92c24f7ca74d7be582ae698fb992404df3"} Jul 09 04:01:09.228949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.228900 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:01:09.230225 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.230201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" event={"ID":"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6","Type":"ContainerStarted","Data":"a749f3a9721411b1eeee60d75bde5fc9d40d8410411bfa1207d70cecbf4576c4"} Jul 09 04:01:09.231923 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.231874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59497797c7-88275" event={"ID":"f89e374d-dac0-4a80-9412-f0e7a94d8a82","Type":"ContainerStarted","Data":"cabb3c48228aa3d2742512f307e17cbd56ec0d167b8f4c3206fce0fe70815074"} Jul 09 04:01:09.231923 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.231920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59497797c7-88275" event={"ID":"f89e374d-dac0-4a80-9412-f0e7a94d8a82","Type":"ContainerStarted","Data":"fa677383c49fcd57e1b82706a9bac5c3b23649aff3753bbe9b102d52771c69d6"} Jul 09 04:01:09.233747 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.233701 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5s6xd" event={"ID":"94ba1740-99fb-47a0-9c83-0689c076cc34","Type":"ContainerStarted","Data":"13cd9c107fd1a4422668d2c46843a0d79cab4b1979160e4c1e3ec1d7fea3e195"} Jul 09 04:01:09.237729 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.237684 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" podStartSLOduration=68.23766781 podStartE2EDuration="1m8.23766781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:09.237314324 +0000 UTC m=+68.841273078" watchObservedRunningTime="2026-07-09 04:01:09.23766781 +0000 UTC m=+68.841626552" Jul 09 04:01:09.254412 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.254297 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59497797c7-88275" podStartSLOduration=66.254285809 podStartE2EDuration="1m6.254285809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:09.252857902 +0000 UTC m=+68.856816643" watchObservedRunningTime="2026-07-09 04:01:09.254285809 +0000 UTC m=+68.858244555" Jul 09 04:01:09.266278 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.266243 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vfs25" podStartSLOduration=68.266232711 podStartE2EDuration="1m8.266232711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:09.265697081 +0000 UTC m=+68.869655823" watchObservedRunningTime="2026-07-09 04:01:09.266232711 +0000 UTC m=+68.870191478" Jul 09 04:01:09.372985 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.372880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:09.377485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:09.375846 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:10.241930 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:10.241874 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:10.243817 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:10.243625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59497797c7-88275" Jul 09 04:01:10.917060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:10.917025 2570 scope.go:117] "RemoveContainer" containerID="93ac9f808269b295ea1536e5971604986fa54b59e662136b01e8faa9a77cb180" Jul 09 04:01:16.269816 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.269173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-54pxr" event={"ID":"21f4e347-8cf3-4ecc-b6c1-18e9c590e662","Type":"ContainerStarted","Data":"53520e9130f86f6d374f0cf0af22318ce9ca460bcd2725d80265bff85e6cade9"} Jul 09 04:01:16.271603 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.271572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp2dp" event={"ID":"2fd95163-b562-4bd6-9157-c69ebc28e76b","Type":"ContainerStarted","Data":"d3b84cd60bdbd0edd90cd1288f546e57631bf9509d2f4891c12b9f285b2a192b"} Jul 09 04:01:16.273917 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.273892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" event={"ID":"e878bddb-eee6-4794-90be-2db53f9ad7cc","Type":"ContainerStarted","Data":"69684409d5d9e95424d945edc8029b45da23edefb5397d87799e631bb7f9a65b"} Jul 09 04:01:16.276865 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.276625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:01:16.276865 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.276705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" event={"ID":"419714ad-2c57-4966-8ed2-173135fac77d","Type":"ContainerStarted","Data":"5fac36ff53f1806413ea274d6c9adc42d1e1e10d5045d08df3a2598cc18f3f6f"} Jul 09 04:01:16.277101 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.277057 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:01:16.279855 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.279836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pppkm" event={"ID":"b2c5f642-6ed5-442b-8e2c-e74ee36237bc","Type":"ContainerStarted","Data":"a05c81d05bc1a33c98b792af01d08984ecbbab999859cd6895065ffd1d6ac60e"} Jul 09 04:01:16.281824 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.281759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" event={"ID":"91335399-c3f6-414e-9086-0ce6b4f5061c","Type":"ContainerStarted","Data":"5ca5fe5d5b8c38b969869aec7c7181af65900040039ef44a703629f1eec2f4d8"} Jul 09 04:01:16.281824 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.281800 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" event={"ID":"91335399-c3f6-414e-9086-0ce6b4f5061c","Type":"ContainerStarted","Data":"dc602b7e3b305de2959291b3683c7fb3f4595589dcc55088d1eab55ad3fbb1ef"} Jul 09 04:01:16.283773 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.283732 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-54pxr" podStartSLOduration=68.168354158 podStartE2EDuration="1m15.283716112s" totalImagesPullingTime="7.115361954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.283177882 +0000 UTC m=+75.887136640" watchObservedRunningTime="2026-07-09 04:01:16.283716112 +0000 UTC m=+75.887674853" Jul 09 04:01:16.284701 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.284618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" event={"ID":"a0e1b35d-5a64-4eaa-a85c-9899af48c4a6","Type":"ContainerStarted","Data":"f0de9cb2cc6146d6832574a6121aa35c66e2853128a0e86e0793b6c3eaaf6817"} Jul 09 04:01:16.292693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.292656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5s6xd" event={"ID":"94ba1740-99fb-47a0-9c83-0689c076cc34","Type":"ContainerStarted","Data":"2e6eb2bbd486fa6e9c9bd5874fdd472d75ba85ce2e19bcb3dc40c1265842943a"} Jul 09 04:01:16.292858 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.292841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5s6xd" event={"ID":"94ba1740-99fb-47a0-9c83-0689c076cc34","Type":"ContainerStarted","Data":"485be52e9d7bcc01afa3b8861157008b708a1e2054a3b48b7d59c011284f19fd"} Jul 09 04:01:16.293049 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.293033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5s6xd" Jul 09 04:01:16.301031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.300989 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wgp2m" podStartSLOduration=66.280868254 podStartE2EDuration="1m13.300954113s" totalImagesPullingTime="7.020085859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.297828382 +0000 UTC m=+75.901787130" watchObservedRunningTime="2026-07-09 04:01:16.300954113 +0000 UTC m=+75.904912855" Jul 09 04:01:16.316396 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.316229 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pkbg8" podStartSLOduration=62.317484271 podStartE2EDuration="1m9.316211439s" totalImagesPullingTime="6.998727168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.31371644 +0000 UTC m=+75.917675181" watchObservedRunningTime="2026-07-09 04:01:16.316211439 +0000 UTC m=+75.920170182" Jul 09 04:01:16.370465 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.370411 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" podStartSLOduration=63.892515992 podStartE2EDuration="1m13.370390723s" totalImagesPullingTime="9.477874731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.370113628 +0000 UTC m=+75.974072370" watchObservedRunningTime="2026-07-09 04:01:16.370390723 +0000 UTC m=+75.974349465" Jul 09 04:01:16.396027 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.395977 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pppkm" podStartSLOduration=17.43503889 podStartE2EDuration="28.395955984s" totalImagesPullingTime="10.960917094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.39460972 +0000 UTC m=+75.998568478" watchObservedRunningTime="2026-07-09 04:01:16.395955984 +0000 UTC m=+75.999914727" Jul 09 04:01:16.415235 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.415186 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7d4qd" podStartSLOduration=66.354128088 podStartE2EDuration="1m13.415168128s" totalImagesPullingTime="7.06104004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.414412224 +0000 UTC m=+76.018370965" watchObservedRunningTime="2026-07-09 04:01:16.415168128 +0000 UTC m=+76.019126873" Jul 09 04:01:16.440786 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.440413 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5s6xd" podStartSLOduration=34.492269719 podStartE2EDuration="41.440395409s" totalImagesPullingTime="6.94812569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:16.439832015 +0000 UTC m=+76.043790755" watchObservedRunningTime="2026-07-09 04:01:16.440395409 +0000 UTC m=+76.044354148" Jul 09 04:01:16.534306 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.534242 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-jh6p7" Jul 09 04:01:16.606611 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.606573 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm"] Jul 09 04:01:16.629457 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.629430 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm"] Jul 09 04:01:16.629591 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.629520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:16.631658 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.631635 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cmh59\"" Jul 09 04:01:16.631791 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.631688 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 04:01:16.732690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.732658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7fdlm\" (UID: \"7091ee9b-53d4-4196-828a-c44000073418\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:16.833016 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:16.832987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7fdlm\" (UID: \"7091ee9b-53d4-4196-828a-c44000073418\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:16.833153 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:16.833114 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 04:01:16.833199 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:16.833170 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates podName:7091ee9b-53d4-4196-828a-c44000073418 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:17.333153209 +0000 UTC m=+76.937111928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-7fdlm" (UID: "7091ee9b-53d4-4196-828a-c44000073418") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 04:01:17.298070 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:17.298028 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp2dp" event={"ID":"2fd95163-b562-4bd6-9157-c69ebc28e76b","Type":"ContainerStarted","Data":"ddec4001eb409b4be722970cc32fd35abf82bacef5d103f6f0ae9b3c54dd7e23"} Jul 09 04:01:17.314147 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:17.314102 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xp2dp" podStartSLOduration=69.197326168 podStartE2EDuration="1m16.314088594s" totalImagesPullingTime="7.116762426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:17.31305369 +0000 UTC m=+76.917012447" watchObservedRunningTime="2026-07-09 04:01:17.314088594 +0000 UTC m=+76.918047334" Jul 09 04:01:17.336786 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:17.336762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7fdlm\" (UID: \"7091ee9b-53d4-4196-828a-c44000073418\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:17.336871 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:17.336851 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 04:01:17.336922 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:17.336905 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates podName:7091ee9b-53d4-4196-828a-c44000073418 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:18.336893478 +0000 UTC m=+77.940852197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-7fdlm" (UID: "7091ee9b-53d4-4196-828a-c44000073418") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 04:01:18.343596 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:18.343566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7fdlm\" (UID: \"7091ee9b-53d4-4196-828a-c44000073418\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:18.345693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:18.345674 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7091ee9b-53d4-4196-828a-c44000073418-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7fdlm\" (UID: \"7091ee9b-53d4-4196-828a-c44000073418\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:18.438796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:18.438770 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:18.585899 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:18.585871 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm"] Jul 09 04:01:18.588378 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:18.588324 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091ee9b_53d4_4196_828a_c44000073418.slice/crio-c36b03a3e2b40af84f79067a1df7e3275cebd242ff0f973c01e1323e6f13ca4a WatchSource:0}: Error finding container c36b03a3e2b40af84f79067a1df7e3275cebd242ff0f973c01e1323e6f13ca4a: Status 404 returned error can't find the container with id c36b03a3e2b40af84f79067a1df7e3275cebd242ff0f973c01e1323e6f13ca4a Jul 09 04:01:19.306186 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:19.306151 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" event={"ID":"7091ee9b-53d4-4196-828a-c44000073418","Type":"ContainerStarted","Data":"c36b03a3e2b40af84f79067a1df7e3275cebd242ff0f973c01e1323e6f13ca4a"} Jul 09 04:01:21.313839 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:21.313800 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" event={"ID":"7091ee9b-53d4-4196-828a-c44000073418","Type":"ContainerStarted","Data":"3573d935ee789fbb79036f458b26293fdb35dbcbfa7658671546ac1fa33d1770"} Jul 09 04:01:21.314183 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:21.313880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:21.318570 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:21.318549 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" Jul 09 04:01:21.327795 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:21.327753 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7fdlm" podStartSLOduration=3.591079362 podStartE2EDuration="5.327740031s" totalImagesPullingTime="1.736660669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:21.326519257 +0000 UTC m=+80.930477999" watchObservedRunningTime="2026-07-09 04:01:21.327740031 +0000 UTC m=+80.931698808" Jul 09 04:01:25.241927 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.241803 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:01:25.244645 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.244623 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.246737 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.246712 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 04:01:25.247308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247278 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 04:01:25.247427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247304 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 04:01:25.247427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 04:01:25.247427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd6vl\"" Jul 09 04:01:25.247427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247385 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 04:01:25.247427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247407 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 04:01:25.247719 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.247703 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 04:01:25.255484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.255463 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:01:25.394747 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.394847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394749 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.394847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pksm\" (UniqueName: \"kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.394847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.394962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.394962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.394898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pksm\" (UniqueName: \"kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496518 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496857 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496933 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496911 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.496986 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.496966 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.498575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.498555 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.498652 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.498622 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.503823 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.503801 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pksm\" (UniqueName: \"kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm\") pod \"console-74c65b75c8-zw8j8\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.553670 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.553649 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:25.677392 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:25.677369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:01:25.679848 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:25.679824 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b056f69_9a6b_4cdf_915e_bcc0ee781c5a.slice/crio-6c903717d8f0a616b6bd7e7029dfa67308d3b3a758a83f6ae359fa675eacfe03 WatchSource:0}: Error finding container 6c903717d8f0a616b6bd7e7029dfa67308d3b3a758a83f6ae359fa675eacfe03: Status 404 returned error can't find the container with id 6c903717d8f0a616b6bd7e7029dfa67308d3b3a758a83f6ae359fa675eacfe03 Jul 09 04:01:26.332448 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:26.332410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c65b75c8-zw8j8" event={"ID":"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a","Type":"ContainerStarted","Data":"6c903717d8f0a616b6bd7e7029dfa67308d3b3a758a83f6ae359fa675eacfe03"} Jul 09 04:01:27.304764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:27.304728 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5s6xd" Jul 09 04:01:27.384778 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:27.384747 2570 patch_prober.go:28] interesting pod/image-registry-79cd586689-zfs5d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 04:01:27.385123 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:27.384794 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 04:01:29.347365 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:29.347319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c65b75c8-zw8j8" event={"ID":"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a","Type":"ContainerStarted","Data":"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5"} Jul 09 04:01:29.364296 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:29.364245 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c65b75c8-zw8j8" podStartSLOduration=1.093112273 podStartE2EDuration="4.364230608s" totalImagesPullingTime="3.271118335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:29.363393469 +0000 UTC m=+88.967352220" watchObservedRunningTime="2026-07-09 04:01:29.364230608 +0000 UTC m=+88.968189348" Jul 09 04:01:30.168731 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.168702 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mn96m"] Jul 09 04:01:30.171096 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.171080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.173678 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.173654 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 04:01:30.173780 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.173761 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 04:01:30.174068 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.174050 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wpfxj\"" Jul 09 04:01:30.174616 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.174599 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 04:01:30.174852 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.174840 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 04:01:30.245468 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.245448 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:30.336149 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336259 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336158 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-textfile\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336309 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336267 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-root\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336375 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-accelerators-collector-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336375 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-metrics-client-ca\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336477 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdjk\" (UniqueName: \"kubernetes.io/projected/8de4c8f0-c912-43fb-8542-f884c6f58243-kube-api-access-qjdjk\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336531 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-sys\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336583 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.336583 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.336572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-wtmp\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.437865 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.437801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-root\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.437865 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.437828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-accelerators-collector-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.437913 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-root\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-metrics-client-ca\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdjk\" (UniqueName: \"kubernetes.io/projected/8de4c8f0-c912-43fb-8542-f884c6f58243-kube-api-access-qjdjk\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-sys\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438228 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-accelerators-collector-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-wtmp\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438315 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:30.438305 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-textfile\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438406 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-sys\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:30.438433 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls podName:8de4c8f0-c912-43fb-8542-f884c6f58243 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:30.938411618 +0000 UTC m=+90.542370348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls") pod "node-exporter-mn96m" (UID: "8de4c8f0-c912-43fb-8542-f884c6f58243") : secret "node-exporter-tls" not found Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438440 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-wtmp\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438536 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de4c8f0-c912-43fb-8542-f884c6f58243-metrics-client-ca\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.438710 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.438614 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-textfile\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.440588 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.440567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.448820 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.448788 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdjk\" (UniqueName: \"kubernetes.io/projected/8de4c8f0-c912-43fb-8542-f884c6f58243-kube-api-access-qjdjk\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.942779 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.942752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:30.944916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:30.944896 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8de4c8f0-c912-43fb-8542-f884c6f58243-node-exporter-tls\") pod \"node-exporter-mn96m\" (UID: \"8de4c8f0-c912-43fb-8542-f884c6f58243\") " pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:31.080219 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.080187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mn96m" Jul 09 04:01:31.092869 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:31.092842 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de4c8f0_c912_43fb_8542_f884c6f58243.slice/crio-746d6ff5531cc837e74a91060717c8ccb952b680e124a8f1119c9e7b4ea9d988 WatchSource:0}: Error finding container 746d6ff5531cc837e74a91060717c8ccb952b680e124a8f1119c9e7b4ea9d988: Status 404 returned error can't find the container with id 746d6ff5531cc837e74a91060717c8ccb952b680e124a8f1119c9e7b4ea9d988 Jul 09 04:01:31.100030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.100010 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:01:31.102934 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.102917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.105649 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.105622 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 04:01:31.106224 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.106203 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 04:01:31.106313 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.106270 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 04:01:31.106431 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.106412 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 04:01:31.106569 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.106553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 04:01:31.106676 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.106625 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 04:01:31.107049 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.107017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 04:01:31.107176 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.107073 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 04:01:31.107283 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.107265 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m4htm\"" Jul 09 04:01:31.107364 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.107269 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 04:01:31.125613 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.125593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:01:31.245095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245056 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245210 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztlm\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245300 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245545 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245545 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.245545 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.245444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.345958 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.345932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.345966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.345993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztlm\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.346449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.346409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.347124 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.347083 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.347894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.347867 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.348852 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.348827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.349481 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.349462 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.350297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.349915 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.350297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.350262 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.350297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.350292 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.350977 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.350599 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.351071 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.350997 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.351143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.351129 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.351188 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.351161 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.351642 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.351620 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.357393 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.357344 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztlm\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm\") pod \"alertmanager-main-0\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.360222 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.360199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mn96m" event={"ID":"8de4c8f0-c912-43fb-8542-f884c6f58243","Type":"ContainerStarted","Data":"746d6ff5531cc837e74a91060717c8ccb952b680e124a8f1119c9e7b4ea9d988"} Jul 09 04:01:31.412719 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.412698 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:01:31.546990 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:31.546953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:01:31.549207 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:31.549179 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b0764_5261_4fa2_b9e9_72e299e42dd1.slice/crio-052e63c7795e03a1c9704641f48b29149184db63c0060e7bdf2db47d17c2ff45 WatchSource:0}: Error finding container 052e63c7795e03a1c9704641f48b29149184db63c0060e7bdf2db47d17c2ff45: Status 404 returned error can't find the container with id 052e63c7795e03a1c9704641f48b29149184db63c0060e7bdf2db47d17c2ff45 Jul 09 04:01:32.116259 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.116208 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77b5c685d9-gn84z"] Jul 09 04:01:32.125641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.125611 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.128320 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128292 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 04:01:32.128472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128450 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5w85x\"" Jul 09 04:01:32.128638 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128603 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 04:01:32.129111 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128877 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 04:01:32.129111 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128903 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5lotggl50kn7e\"" Jul 09 04:01:32.129111 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.128909 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 04:01:32.129111 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.129107 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 04:01:32.133665 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.133644 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77b5c685d9-gn84z"] Jul 09 04:01:32.253074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253196 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253196 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253196 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7tz\" (UniqueName: \"kubernetes.io/projected/0b891226-48ff-4e86-84bd-610185b43eed-kube-api-access-xr7tz\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-grpc-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253247 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b891226-48ff-4e86-84bd-610185b43eed-metrics-client-ca\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.253475 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.253324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354167 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-grpc-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b891226-48ff-4e86-84bd-610185b43eed-metrics-client-ca\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354524 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354524 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.354524 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.354378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7tz\" (UniqueName: \"kubernetes.io/projected/0b891226-48ff-4e86-84bd-610185b43eed-kube-api-access-xr7tz\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.355057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.355026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b891226-48ff-4e86-84bd-610185b43eed-metrics-client-ca\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.356700 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.356669 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.357014 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.356992 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-grpc-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.357062 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.357016 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-tls\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.357062 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.356994 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.357062 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.356998 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.357159 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.357081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b891226-48ff-4e86-84bd-610185b43eed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.362766 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.362748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7tz\" (UniqueName: \"kubernetes.io/projected/0b891226-48ff-4e86-84bd-610185b43eed-kube-api-access-xr7tz\") pod \"thanos-querier-77b5c685d9-gn84z\" (UID: \"0b891226-48ff-4e86-84bd-610185b43eed\") " pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.365438 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.365411 2570 generic.go:356] "Generic (PLEG): container finished" podID="8de4c8f0-c912-43fb-8542-f884c6f58243" containerID="8000b0630b5a8761386dfc854beccc890c1f3bba8110f46093d10d4d79315776" exitCode=0 Jul 09 04:01:32.365523 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.365490 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mn96m" event={"ID":"8de4c8f0-c912-43fb-8542-f884c6f58243","Type":"ContainerDied","Data":"8000b0630b5a8761386dfc854beccc890c1f3bba8110f46093d10d4d79315776"} Jul 09 04:01:32.366834 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.366781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"052e63c7795e03a1c9704641f48b29149184db63c0060e7bdf2db47d17c2ff45"} Jul 09 04:01:32.438991 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.438959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:32.595246 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:32.595185 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77b5c685d9-gn84z"] Jul 09 04:01:32.768906 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:32.768810 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b891226_48ff_4e86_84bd_610185b43eed.slice/crio-f62a4defa4f6e1ef48c7e9688697eb45d458c693bb38afcc64a411bb0a75d6c8 WatchSource:0}: Error finding container f62a4defa4f6e1ef48c7e9688697eb45d458c693bb38afcc64a411bb0a75d6c8: Status 404 returned error can't find the container with id f62a4defa4f6e1ef48c7e9688697eb45d458c693bb38afcc64a411bb0a75d6c8 Jul 09 04:01:33.372371 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.372322 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f" exitCode=0 Jul 09 04:01:33.372539 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.372376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f"} Jul 09 04:01:33.373687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.373661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"f62a4defa4f6e1ef48c7e9688697eb45d458c693bb38afcc64a411bb0a75d6c8"} Jul 09 04:01:33.378846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.378826 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mn96m" event={"ID":"8de4c8f0-c912-43fb-8542-f884c6f58243","Type":"ContainerStarted","Data":"ad98e95140e20b039192a00c290c9bf97aafd78feea9db1894ba82922d17b978"} Jul 09 04:01:33.378945 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.378862 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mn96m" event={"ID":"8de4c8f0-c912-43fb-8542-f884c6f58243","Type":"ContainerStarted","Data":"78854cca26189b195478b8dee74a7bf57d0dbd332a2eeb3d2cb442b637c5fa01"} Jul 09 04:01:33.437324 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.437283 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mn96m" podStartSLOduration=2.289044712 podStartE2EDuration="3.437269768s" totalImagesPullingTime="1.085670929s" totalInitContainerRuntime="62.554127ms" isStatefulPod=true podCreationTimestamp="2026-07-09 04:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:33.43664058 +0000 UTC m=+93.040599346" watchObservedRunningTime="2026-07-09 04:01:33.437269768 +0000 UTC m=+93.041228509" Jul 09 04:01:33.773912 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:33.773836 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:01:34.532479 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.532442 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f7d77f95f-2phch"] Jul 09 04:01:34.540639 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.540609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543557 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9wfbg\"" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543708 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9hcdob0pohuot\"" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543712 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543781 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543885 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 09 04:01:34.544074 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.543973 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 04:01:34.554068 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.554032 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7d77f95f-2phch"] Jul 09 04:01:34.674115 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-client-certs\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhljb\" (UniqueName: \"kubernetes.io/projected/fad11bd7-6357-4ae2-8966-8684d2ad24b7-kube-api-access-qhljb\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fad11bd7-6357-4ae2-8966-8684d2ad24b7-audit-log\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-client-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674327 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-metrics-server-audit-profiles\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.674460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.674442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-tls\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-client-certs\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhljb\" (UniqueName: \"kubernetes.io/projected/fad11bd7-6357-4ae2-8966-8684d2ad24b7-kube-api-access-qhljb\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fad11bd7-6357-4ae2-8966-8684d2ad24b7-audit-log\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-client-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-metrics-server-audit-profiles\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.776989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.776447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-tls\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.777911 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.777163 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fad11bd7-6357-4ae2-8966-8684d2ad24b7-audit-log\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.778034 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.777906 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.778827 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.778767 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fad11bd7-6357-4ae2-8966-8684d2ad24b7-metrics-server-audit-profiles\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.783444 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.783314 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-client-ca-bundle\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.783884 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.783833 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-client-certs\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.785378 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.785301 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fad11bd7-6357-4ae2-8966-8684d2ad24b7-secret-metrics-server-tls\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.792487 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.791724 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhljb\" (UniqueName: \"kubernetes.io/projected/fad11bd7-6357-4ae2-8966-8684d2ad24b7-kube-api-access-qhljb\") pod \"metrics-server-f7d77f95f-2phch\" (UID: \"fad11bd7-6357-4ae2-8966-8684d2ad24b7\") " pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.807511 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.807465 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:01:34.839786 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.839750 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:01:34.843737 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.843708 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.853206 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.852787 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 04:01:34.853206 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.852793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:01:34.860093 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.860066 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:34.979056 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979238 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979084 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979238 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979385 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppmg\" (UniqueName: \"kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979385 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979385 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:34.979541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:34.979441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080427 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080650 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppmg\" (UniqueName: \"kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.080949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.080791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.081788 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.081193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.081788 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.081533 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.082000 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.081976 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.082072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.081976 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.084743 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.084721 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.085145 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.085127 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.090688 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.090664 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppmg\" (UniqueName: \"kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg\") pod \"console-655f495bf5-ntv4t\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.160665 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.160633 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:35.286401 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.274617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7d77f95f-2phch"] Jul 09 04:01:35.362150 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.362122 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:01:35.390890 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.390838 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed"} Jul 09 04:01:35.392287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.392247 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" event={"ID":"fad11bd7-6357-4ae2-8966-8684d2ad24b7","Type":"ContainerStarted","Data":"b65e0e9d75b01f15389c7946722060f524309341eac7d9e27f123b7aacf28311"} Jul 09 04:01:35.554108 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:35.554069 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:01:35.837363 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:35.837290 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3fa775_c11a_4aed_95bd_c5a6fc516933.slice/crio-e1e47a8a8d741a73dd6221c9023c97cdacd549bf79f484d986a00d46ddaf765b WatchSource:0}: Error finding container e1e47a8a8d741a73dd6221c9023c97cdacd549bf79f484d986a00d46ddaf765b: Status 404 returned error can't find the container with id e1e47a8a8d741a73dd6221c9023c97cdacd549bf79f484d986a00d46ddaf765b Jul 09 04:01:36.277077 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.277028 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:01:36.288094 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.288063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.290736 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.290707 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 04:01:36.291247 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291224 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 04:01:36.291591 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291564 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h98tj\"" Jul 09 04:01:36.291725 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291702 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 04:01:36.291823 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 04:01:36.291898 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291818 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 04:01:36.291898 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.291852 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 04:01:36.292037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292029 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 04:01:36.292135 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292113 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 04:01:36.292201 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292169 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 04:01:36.292260 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292238 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 04:01:36.292309 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292292 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 04:01:36.292406 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292390 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 04:01:36.292845 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292787 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 04:01:36.292998 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.292888 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-brsfh10uukuca\"" Jul 09 04:01:36.295069 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.295046 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 04:01:36.298489 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.298469 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:01:36.400481 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.400385 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88"} Jul 09 04:01:36.400481 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.400452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba"} Jul 09 04:01:36.400481 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.400479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780"} Jul 09 04:01:36.400725 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.400512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f"} Jul 09 04:01:36.402407 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.402376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"1250507353b29a6fc49101f12221069134bd9270a78db89a56e06ccb9d8d0902"} Jul 09 04:01:36.402526 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.402442 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"b001efb3ab0c4617fb6b8c8a0a7ce3b76733cff6c3ab94844d2bccc2ee5b8b83"} Jul 09 04:01:36.402526 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.402479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"3dc79d26a30193d37d1e008156d1283491964839d41a3452433b76c78ca1b0a7"} Jul 09 04:01:36.403798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655f495bf5-ntv4t" event={"ID":"ac3fa775-c11a-4aed-95bd-c5a6fc516933","Type":"ContainerStarted","Data":"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e"} Jul 09 04:01:36.403916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403815 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655f495bf5-ntv4t" event={"ID":"ac3fa775-c11a-4aed-95bd-c5a6fc516933","Type":"ContainerStarted","Data":"e1e47a8a8d741a73dd6221c9023c97cdacd549bf79f484d986a00d46ddaf765b"} Jul 09 04:01:36.403916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.403916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.403916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404082 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404082 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.403980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404219 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404249 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404407 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404456 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404505 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404538 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404615 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.404713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.404710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.423323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.423269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-655f495bf5-ntv4t" podStartSLOduration=2.423247577 podStartE2EDuration="2.423247577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:36.421809982 +0000 UTC m=+96.025768738" watchObservedRunningTime="2026-07-09 04:01:36.423247577 +0000 UTC m=+96.027206319" Jul 09 04:01:36.505825 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.505789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.505834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.505878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.505905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.505932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506299 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.506244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506299 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.506285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506474 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.506311 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506474 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.506388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.506474 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.506420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507365 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507568 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507823 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.507970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.508714 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.508947 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.508847 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.509800 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.509028 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.509800 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.509702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.510819 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.510791 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.511701 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.511654 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.511791 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.511712 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.511930 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.511908 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.511999 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.511936 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.512438 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.512396 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.512636 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.512615 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.513075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.512985 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.513075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.513015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.513075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.513070 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.513604 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.513566 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.514142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.514107 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.515225 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.515201 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47\") pod \"prometheus-k8s-0\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.604617 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.604585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:36.762460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:36.762423 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:01:37.199729 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:37.199689 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2122e852_0f7f_44b0_a58d_02b85e66470c.slice/crio-ac4e86037c3d85fd6d19816b3c35e4b1954733f46689b873467ba0b35142e046 WatchSource:0}: Error finding container ac4e86037c3d85fd6d19816b3c35e4b1954733f46689b873467ba0b35142e046: Status 404 returned error can't find the container with id ac4e86037c3d85fd6d19816b3c35e4b1954733f46689b873467ba0b35142e046 Jul 09 04:01:37.409491 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:37.409462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"ac4e86037c3d85fd6d19816b3c35e4b1954733f46689b873467ba0b35142e046"} Jul 09 04:01:38.416446 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.416409 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" exitCode=0 Jul 09 04:01:38.416874 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.416496 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} Jul 09 04:01:38.420494 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.420466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerStarted","Data":"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332"} Jul 09 04:01:38.423871 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.423841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"7c044a7c1d236f792b6d1cba9e31d0b90252afda1794db955405b6e534ff3419"} Jul 09 04:01:38.424000 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.423922 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"1229cda99f7b7dd143d0832c90239d179de1a902f54ebcd77547960bafdb318c"} Jul 09 04:01:38.424000 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.423963 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" event={"ID":"0b891226-48ff-4e86-84bd-610185b43eed","Type":"ContainerStarted","Data":"c1b0b8430a2f1321a3bf611b27dd9cae449dc5779b8957b447fb4752325ba0cd"} Jul 09 04:01:38.424117 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.424015 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:38.425913 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.425882 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" event={"ID":"fad11bd7-6357-4ae2-8966-8684d2ad24b7","Type":"ContainerStarted","Data":"89874c42e4b379f6ae1a1db4f38663fc1baa4bf8d6abfc93768b032303be6904"} Jul 09 04:01:38.466277 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.466208 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" podStartSLOduration=2.050854422 podStartE2EDuration="6.46619142s" totalImagesPullingTime="4.415336998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:38.464567071 +0000 UTC m=+98.068525825" watchObservedRunningTime="2026-07-09 04:01:38.46619142 +0000 UTC m=+98.070150161" Jul 09 04:01:38.490797 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.490742 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.129798839 podStartE2EDuration="7.490723324s" totalImagesPullingTime="4.344828456s" totalInitContainerRuntime="16.096029ms" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:38.490198411 +0000 UTC m=+98.094157470" watchObservedRunningTime="2026-07-09 04:01:38.490723324 +0000 UTC m=+98.094682064" Jul 09 04:01:38.509844 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:38.509784 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" podStartSLOduration=2.41019613 podStartE2EDuration="4.509766099s" totalImagesPullingTime="2.099569969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:38.508257618 +0000 UTC m=+98.112216395" watchObservedRunningTime="2026-07-09 04:01:38.509766099 +0000 UTC m=+98.113724839" Jul 09 04:01:40.243871 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:40.243836 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vfs25" Jul 09 04:01:41.083928 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.083891 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:01:41.114510 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.114479 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:01:41.119169 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.119140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.132091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.132061 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:01:41.150899 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.150861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.150911 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.150959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgxm\" (UniqueName: \"kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.151027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151264 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.151139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151264 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.151174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.151264 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.151199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.251989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.252835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.253051 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.252859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgxm\" (UniqueName: \"kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.253806 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.253782 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.253934 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.253854 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.254131 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.254104 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.254235 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.254139 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.255016 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.254988 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.256225 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.256202 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.260819 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.260796 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgxm\" (UniqueName: \"kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm\") pod \"console-7fc85c95fb-74r9c\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.433031 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.432924 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:41.907999 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:41.907974 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:01:41.911388 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:01:41.911336 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf7fc69_8244_460c_b615_edb157dc7589.slice/crio-3a04e0959ce187724e130a25ff7a2404c60498e950d0418d1272a28fbcf2030b WatchSource:0}: Error finding container 3a04e0959ce187724e130a25ff7a2404c60498e950d0418d1272a28fbcf2030b: Status 404 returned error can't find the container with id 3a04e0959ce187724e130a25ff7a2404c60498e950d0418d1272a28fbcf2030b Jul 09 04:01:42.451533 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451494 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} Jul 09 04:01:42.452007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} Jul 09 04:01:42.452007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451633 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} Jul 09 04:01:42.452007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451669 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} Jul 09 04:01:42.452007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} Jul 09 04:01:42.452007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.451726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerStarted","Data":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} Jul 09 04:01:42.453207 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.453182 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc85c95fb-74r9c" event={"ID":"bdf7fc69-8244-460c-b615-edb157dc7589","Type":"ContainerStarted","Data":"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c"} Jul 09 04:01:42.453316 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.453225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc85c95fb-74r9c" event={"ID":"bdf7fc69-8244-460c-b615-edb157dc7589","Type":"ContainerStarted","Data":"3a04e0959ce187724e130a25ff7a2404c60498e950d0418d1272a28fbcf2030b"} Jul 09 04:01:42.477565 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.477514 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.039541853 podStartE2EDuration="6.477496993s" totalImagesPullingTime="3.415170817s" totalInitContainerRuntime="22.784323ms" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:42.476919636 +0000 UTC m=+102.080878417" watchObservedRunningTime="2026-07-09 04:01:42.477496993 +0000 UTC m=+102.081455753" Jul 09 04:01:42.493299 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:42.493256 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fc85c95fb-74r9c" podStartSLOduration=1.493241339 podStartE2EDuration="1.493241339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:42.493220957 +0000 UTC m=+102.097179699" watchObservedRunningTime="2026-07-09 04:01:42.493241339 +0000 UTC m=+102.097200079" Jul 09 04:01:44.438126 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:44.438095 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77b5c685d9-gn84z" Jul 09 04:01:45.161285 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:45.161243 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:01:46.605174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:46.605129 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:01:51.433489 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:51.433432 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:51.433489 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:51.433509 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:51.438539 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:51.438513 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:51.506424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:51.506396 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:01:52.508740 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:52.508704 2570 generic.go:356] "Generic (PLEG): container finished" podID="14d79e88-ae22-4f3c-8414-fbaca11408b2" containerID="fd167fc4b85bf0cc91fee106f11cff576f233e968b4f05611a4ff88a3498ce3e" exitCode=0 Jul 09 04:01:52.509164 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:52.508777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" event={"ID":"14d79e88-ae22-4f3c-8414-fbaca11408b2","Type":"ContainerDied","Data":"fd167fc4b85bf0cc91fee106f11cff576f233e968b4f05611a4ff88a3498ce3e"} Jul 09 04:01:52.509278 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:52.509252 2570 scope.go:117] "RemoveContainer" containerID="fd167fc4b85bf0cc91fee106f11cff576f233e968b4f05611a4ff88a3498ce3e" Jul 09 04:01:53.516743 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:53.516705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k7n5c" event={"ID":"14d79e88-ae22-4f3c-8414-fbaca11408b2","Type":"ContainerStarted","Data":"a1a44411a4e0228a56b95f073796a2b1d0ca441a9452ee594b2326edef567129"} Jul 09 04:01:53.518097 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:53.518069 2570 generic.go:356] "Generic (PLEG): container finished" podID="ce0cf1d9-9b4a-4dae-9333-0973ae5928d3" containerID="395476b55c72a5f292e20e4dee348c453e5d399192164387e280872e7a70227f" exitCode=0 Jul 09 04:01:53.518225 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:53.518108 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" event={"ID":"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3","Type":"ContainerDied","Data":"395476b55c72a5f292e20e4dee348c453e5d399192164387e280872e7a70227f"} Jul 09 04:01:53.518457 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:53.518440 2570 scope.go:117] "RemoveContainer" containerID="395476b55c72a5f292e20e4dee348c453e5d399192164387e280872e7a70227f" Jul 09 04:01:54.524862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:54.524825 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wc4zl" event={"ID":"ce0cf1d9-9b4a-4dae-9333-0973ae5928d3","Type":"ContainerStarted","Data":"ce2e756e4da10bd6f2270d9dfdf5d5c491d625d22d0e41c8d42793e61909f6d4"} Jul 09 04:01:54.860642 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:54.860600 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:54.860856 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:54.860683 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:01:58.800467 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:58.800399 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerName="registry" containerID="cri-o://439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6" gracePeriod=30 Jul 09 04:01:59.052584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.052516 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:59.143055 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143251 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143077 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143251 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143117 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143251 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143147 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143448 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143398 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxwxx\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143513 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143449 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143513 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143488 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143615 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143502 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:01:59.143615 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143530 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") pod \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\" (UID: \"be11c46e-b7a1-4194-b71f-6c59140f8c1d\") " Jul 09 04:01:59.143713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143668 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:01:59.143984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143956 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-certificates\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.143984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.143985 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be11c46e-b7a1-4194-b71f-6c59140f8c1d-trusted-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.145826 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.145800 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:01:59.145973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.145948 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:01:59.145973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.145957 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:01:59.146112 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.146024 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx" (OuterVolumeSpecName: "kube-api-access-zxwxx") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "kube-api-access-zxwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:01:59.146150 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.146112 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:01:59.151838 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.151801 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be11c46e-b7a1-4194-b71f-6c59140f8c1d" (UID: "be11c46e-b7a1-4194-b71f-6c59140f8c1d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:01:59.244750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244704 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-registry-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.244750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244742 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-installation-pull-secrets\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.244750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244755 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be11c46e-b7a1-4194-b71f-6c59140f8c1d-image-registry-private-configuration\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.244750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244764 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwxx\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-kube-api-access-zxwxx\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.245012 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244773 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be11c46e-b7a1-4194-b71f-6c59140f8c1d-ca-trust-extracted\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.245012 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.244783 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be11c46e-b7a1-4194-b71f-6c59140f8c1d-bound-sa-token\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:01:59.551449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.551414 2570 generic.go:356] "Generic (PLEG): container finished" podID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerID="439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6" exitCode=0 Jul 09 04:01:59.551621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.551458 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" Jul 09 04:01:59.551621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.551505 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" event={"ID":"be11c46e-b7a1-4194-b71f-6c59140f8c1d","Type":"ContainerDied","Data":"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6"} Jul 09 04:01:59.551621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.551570 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cd586689-zfs5d" event={"ID":"be11c46e-b7a1-4194-b71f-6c59140f8c1d","Type":"ContainerDied","Data":"a82b7d5cd60a0df1113d6ef2a35e894f6aeae72022ef3e7b4afc89a7a8e4a7e4"} Jul 09 04:01:59.551621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.551592 2570 scope.go:117] "RemoveContainer" containerID="439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6" Jul 09 04:01:59.561122 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.561101 2570 scope.go:117] "RemoveContainer" containerID="439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6" Jul 09 04:01:59.561437 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:01:59.561415 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6\": container with ID starting with 439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6 not found: ID does not exist" containerID="439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6" Jul 09 04:01:59.561494 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.561447 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6"} err="failed to get container status \"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6\": rpc error: code = NotFound desc = could not find container \"439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6\": container with ID starting with 439835a71a4eff0150c6e9f6e4d71a3b56ec8be182944617a91ebf99d401dfa6 not found: ID does not exist" Jul 09 04:01:59.572231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.572200 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:01:59.577263 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.577239 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79cd586689-zfs5d"] Jul 09 04:01:59.833147 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:01:59.833058 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c65b75c8-zw8j8" podUID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" containerName="console" containerID="cri-o://27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5" gracePeriod=15 Jul 09 04:02:00.094504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.094440 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:02:00.254202 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254170 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254202 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254207 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254247 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254368 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pksm\" (UniqueName: \"kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254403 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254454 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert\") pod \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\" (UID: \"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a\") " Jul 09 04:02:00.254708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254604 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config" (OuterVolumeSpecName: "console-config") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:00.254798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254769 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:00.254972 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254801 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.254972 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.254930 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:00.256529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.256500 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:00.256529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.256514 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:00.256646 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.256621 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm" (OuterVolumeSpecName: "kube-api-access-9pksm") pod "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" (UID: "4b056f69-9a6b-4cdf-915e-bcc0ee781c5a"). InnerVolumeSpecName "kube-api-access-9pksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:00.355866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.355780 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-oauth-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.355866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.355810 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-console-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.355866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.355822 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pksm\" (UniqueName: \"kubernetes.io/projected/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-kube-api-access-9pksm\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.355866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.355832 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.355866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.355841 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a-oauth-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:00.557235 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.557201 2570 generic.go:356] "Generic (PLEG): container finished" podID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" containerID="27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5" exitCode=0 Jul 09 04:02:00.557447 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.557258 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c65b75c8-zw8j8" Jul 09 04:02:00.557447 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.557293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c65b75c8-zw8j8" event={"ID":"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a","Type":"ContainerDied","Data":"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5"} Jul 09 04:02:00.557447 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.557376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c65b75c8-zw8j8" event={"ID":"4b056f69-9a6b-4cdf-915e-bcc0ee781c5a","Type":"ContainerDied","Data":"6c903717d8f0a616b6bd7e7029dfa67308d3b3a758a83f6ae359fa675eacfe03"} Jul 09 04:02:00.557447 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.557406 2570 scope.go:117] "RemoveContainer" containerID="27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5" Jul 09 04:02:00.566584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.566563 2570 scope.go:117] "RemoveContainer" containerID="27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5" Jul 09 04:02:00.566836 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:00.566813 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5\": container with ID starting with 27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5 not found: ID does not exist" containerID="27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5" Jul 09 04:02:00.566901 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.566846 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5"} err="failed to get container status \"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5\": rpc error: code = NotFound desc = could not find container \"27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5\": container with ID starting with 27f66ef4224260447e63e44e3d688ca8d25d99919d3c3c2ecc2b6c8766f18ba5 not found: ID does not exist" Jul 09 04:02:00.577051 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.577022 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:02:00.580306 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.580275 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c65b75c8-zw8j8"] Jul 09 04:02:00.924691 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.922077 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" path="/var/lib/kubelet/pods/4b056f69-9a6b-4cdf-915e-bcc0ee781c5a/volumes" Jul 09 04:02:00.924691 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:00.922745 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" path="/var/lib/kubelet/pods/be11c46e-b7a1-4194-b71f-6c59140f8c1d/volumes" Jul 09 04:02:01.564382 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:01.564325 2570 generic.go:356] "Generic (PLEG): container finished" podID="d09134d4-951d-4b4a-b251-a450d3176fad" containerID="32f7d29c757e49697965989a9b5098922e527827b9e134d589035b8b3729858d" exitCode=0 Jul 09 04:02:01.564566 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:01.564392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" event={"ID":"d09134d4-951d-4b4a-b251-a450d3176fad","Type":"ContainerDied","Data":"32f7d29c757e49697965989a9b5098922e527827b9e134d589035b8b3729858d"} Jul 09 04:02:01.564800 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:01.564783 2570 scope.go:117] "RemoveContainer" containerID="32f7d29c757e49697965989a9b5098922e527827b9e134d589035b8b3729858d" Jul 09 04:02:02.570469 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:02.570431 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-5fjkw" event={"ID":"d09134d4-951d-4b4a-b251-a450d3176fad","Type":"ContainerStarted","Data":"af45655a90d2a4ad0da8093aa7d3f84b7c59dc100ded1936934383c00a799371"} Jul 09 04:02:06.109221 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.109107 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-655f495bf5-ntv4t" podUID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" containerName="console" containerID="cri-o://e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e" gracePeriod=15 Jul 09 04:02:06.420304 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.420277 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:02:06.522821 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.522783 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.522821 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.522820 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vppmg\" (UniqueName: \"kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.522850 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523022 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523072 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523152 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523143 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:06.523424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523175 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config" (OuterVolumeSpecName: "console-config") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:06.523424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523207 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert\") pod \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\" (UID: \"ac3fa775-c11a-4aed-95bd-c5a6fc516933\") " Jul 09 04:02:06.523424 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523400 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:06.523682 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523657 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-trusted-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.523746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523690 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.523746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523709 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-oauth-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.523746 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.523723 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:06.525180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.525154 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:06.525326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.525301 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg" (OuterVolumeSpecName: "kube-api-access-vppmg") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "kube-api-access-vppmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:06.525469 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.525451 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac3fa775-c11a-4aed-95bd-c5a6fc516933" (UID: "ac3fa775-c11a-4aed-95bd-c5a6fc516933"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:06.594946 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.594901 2570 generic.go:356] "Generic (PLEG): container finished" podID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" containerID="e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e" exitCode=0 Jul 09 04:02:06.595142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.594966 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655f495bf5-ntv4t" Jul 09 04:02:06.595142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.594975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655f495bf5-ntv4t" event={"ID":"ac3fa775-c11a-4aed-95bd-c5a6fc516933","Type":"ContainerDied","Data":"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e"} Jul 09 04:02:06.595142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.595050 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655f495bf5-ntv4t" event={"ID":"ac3fa775-c11a-4aed-95bd-c5a6fc516933","Type":"ContainerDied","Data":"e1e47a8a8d741a73dd6221c9023c97cdacd549bf79f484d986a00d46ddaf765b"} Jul 09 04:02:06.595142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.595070 2570 scope.go:117] "RemoveContainer" containerID="e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e" Jul 09 04:02:06.604876 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.604854 2570 scope.go:117] "RemoveContainer" containerID="e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e" Jul 09 04:02:06.605161 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:06.605141 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e\": container with ID starting with e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e not found: ID does not exist" containerID="e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e" Jul 09 04:02:06.605221 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.605171 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e"} err="failed to get container status \"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e\": rpc error: code = NotFound desc = could not find container \"e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e\": container with ID starting with e431bb8c3552a0a2b53e4479f560dbd635f144ed95edc213bce2c4664cd0342e not found: ID does not exist" Jul 09 04:02:06.622889 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.622849 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:02:06.626058 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.626025 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-655f495bf5-ntv4t"] Jul 09 04:02:06.626941 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.626916 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-oauth-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.627115 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.627096 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3fa775-c11a-4aed-95bd-c5a6fc516933-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.627237 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.627223 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3fa775-c11a-4aed-95bd-c5a6fc516933-console-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.627333 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.627320 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppmg\" (UniqueName: \"kubernetes.io/projected/ac3fa775-c11a-4aed-95bd-c5a6fc516933-kube-api-access-vppmg\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:06.920572 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:06.920534 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" path="/var/lib/kubelet/pods/ac3fa775-c11a-4aed-95bd-c5a6fc516933/volumes" Jul 09 04:02:14.866608 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:14.866573 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:02:14.870619 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:14.870593 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f7d77f95f-2phch" Jul 09 04:02:36.605286 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:36.605241 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:36.621101 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:36.621077 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:36.734676 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:36.734648 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:50.465581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465544 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465863 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="alertmanager" containerID="cri-o://e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed" gracePeriod=120 Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465920 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-metric" containerID="cri-o://fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88" gracePeriod=120 Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465959 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="prom-label-proxy" containerID="cri-o://c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332" gracePeriod=120 Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465961 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy" containerID="cri-o://fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba" gracePeriod=120 Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465980 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="config-reloader" containerID="cri-o://d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f" gracePeriod=120 Jul 09 04:02:50.466073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.465937 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-web" containerID="cri-o://9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780" gracePeriod=120 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.782992 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332" exitCode=0 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783020 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88" exitCode=0 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783026 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba" exitCode=0 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783032 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f" exitCode=0 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783038 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed" exitCode=0 Jul 09 04:02:50.783078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332"} Jul 09 04:02:50.783397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783122 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88"} Jul 09 04:02:50.783397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783149 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba"} Jul 09 04:02:50.783397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f"} Jul 09 04:02:50.783397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:50.783198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed"} Jul 09 04:02:51.739564 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.739537 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:02:51.790684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.790647 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerID="9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780" exitCode=0 Jul 09 04:02:51.790845 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.790729 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780"} Jul 09 04:02:51.790845 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.790746 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 04:02:51.790845 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.790828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8f3b0764-5261-4fa2-b9e9-72e299e42dd1","Type":"ContainerDied","Data":"052e63c7795e03a1c9704641f48b29149184db63c0060e7bdf2db47d17c2ff45"} Jul 09 04:02:51.790968 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.790883 2570 scope.go:117] "RemoveContainer" containerID="c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332" Jul 09 04:02:51.803130 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.803102 2570 scope.go:117] "RemoveContainer" containerID="fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88" Jul 09 04:02:51.811337 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.811316 2570 scope.go:117] "RemoveContainer" containerID="fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba" Jul 09 04:02:51.819288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.819263 2570 scope.go:117] "RemoveContainer" containerID="9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780" Jul 09 04:02:51.826788 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.826770 2570 scope.go:117] "RemoveContainer" containerID="d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f" Jul 09 04:02:51.835084 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835164 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835103 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835164 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835131 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kztlm\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835164 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835105 2570 scope.go:117] "RemoveContainer" containerID="e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed" Jul 09 04:02:51.835331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835166 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835210 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835292 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835317 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835512 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835456 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835512 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835493 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835620 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835523 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835620 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835560 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835620 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835647 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config\") pod \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\" (UID: \"8f3b0764-5261-4fa2-b9e9-72e299e42dd1\") " Jul 09 04:02:51.835907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.835883 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:51.836196 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.836168 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:51.837188 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.837161 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:51.839284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.838671 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:51.840918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.840863 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.840918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.840896 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out" (OuterVolumeSpecName: "config-out") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:51.841116 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.841035 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume" (OuterVolumeSpecName: "config-volume") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.841116 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.841056 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.841116 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.841105 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.841277 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.841232 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.841455 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.841434 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm" (OuterVolumeSpecName: "kube-api-access-kztlm") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "kube-api-access-kztlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:51.847034 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.847006 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.851866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.851846 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config" (OuterVolumeSpecName: "web-config") pod "8f3b0764-5261-4fa2-b9e9-72e299e42dd1" (UID: "8f3b0764-5261-4fa2-b9e9-72e299e42dd1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:51.861853 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.861838 2570 scope.go:117] "RemoveContainer" containerID="f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f" Jul 09 04:02:51.869081 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869060 2570 scope.go:117] "RemoveContainer" containerID="c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332" Jul 09 04:02:51.869362 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.869328 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332\": container with ID starting with c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332 not found: ID does not exist" containerID="c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332" Jul 09 04:02:51.869441 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869376 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332"} err="failed to get container status \"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332\": rpc error: code = NotFound desc = could not find container \"c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332\": container with ID starting with c61cf7e8bd4d91d0bab2b4e71d275abe4386f61bc1dd999a16573ca10ba34332 not found: ID does not exist" Jul 09 04:02:51.869441 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869405 2570 scope.go:117] "RemoveContainer" containerID="fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88" Jul 09 04:02:51.869652 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.869635 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88\": container with ID starting with fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88 not found: ID does not exist" containerID="fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88" Jul 09 04:02:51.869696 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869658 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88"} err="failed to get container status \"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88\": rpc error: code = NotFound desc = could not find container \"fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88\": container with ID starting with fc2e30e2b857c0d7e891dae3919295229ce6abd5426079bb6ead793f5f83dc88 not found: ID does not exist" Jul 09 04:02:51.869696 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869676 2570 scope.go:117] "RemoveContainer" containerID="fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba" Jul 09 04:02:51.869914 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.869896 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba\": container with ID starting with fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba not found: ID does not exist" containerID="fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba" Jul 09 04:02:51.869952 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869921 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba"} err="failed to get container status \"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba\": rpc error: code = NotFound desc = could not find container \"fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba\": container with ID starting with fe3b3308ce05dc6db570102d71969d884e53e612d8c8ab80d4bc81d4e48ccbba not found: ID does not exist" Jul 09 04:02:51.869952 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.869937 2570 scope.go:117] "RemoveContainer" containerID="9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780" Jul 09 04:02:51.870138 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.870122 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780\": container with ID starting with 9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780 not found: ID does not exist" containerID="9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780" Jul 09 04:02:51.870182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870141 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780"} err="failed to get container status \"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780\": rpc error: code = NotFound desc = could not find container \"9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780\": container with ID starting with 9dbcb1d2f4a329ba15dad9a842d7900998296baa6828b7f5559ae8eaa0987780 not found: ID does not exist" Jul 09 04:02:51.870182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870154 2570 scope.go:117] "RemoveContainer" containerID="d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f" Jul 09 04:02:51.870328 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.870313 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f\": container with ID starting with d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f not found: ID does not exist" containerID="d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f" Jul 09 04:02:51.870402 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870331 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f"} err="failed to get container status \"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f\": rpc error: code = NotFound desc = could not find container \"d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f\": container with ID starting with d121b5a6d629f0219bc3a13694f9d001561962c4b506cd8bbca1df3a3b27b39f not found: ID does not exist" Jul 09 04:02:51.870402 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870346 2570 scope.go:117] "RemoveContainer" containerID="e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed" Jul 09 04:02:51.870599 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.870582 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed\": container with ID starting with e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed not found: ID does not exist" containerID="e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed" Jul 09 04:02:51.870667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870614 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed"} err="failed to get container status \"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed\": rpc error: code = NotFound desc = could not find container \"e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed\": container with ID starting with e6534242a591172c1d6bce8dca15ec45c88d57ad2c47c81c9ff14d1b3f51d5ed not found: ID does not exist" Jul 09 04:02:51.870667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870637 2570 scope.go:117] "RemoveContainer" containerID="f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f" Jul 09 04:02:51.870911 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:51.870894 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f\": container with ID starting with f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f not found: ID does not exist" containerID="f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f" Jul 09 04:02:51.870956 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.870916 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f"} err="failed to get container status \"f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f\": rpc error: code = NotFound desc = could not find container \"f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f\": container with ID starting with f01dcd773cc59d3f7c6028c8afb96fdedce78543d15a4a56268611373f75408f not found: ID does not exist" Jul 09 04:02:51.936616 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936575 2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-volume\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936616 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936608 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936616 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936619 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kztlm\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-kube-api-access-kztlm\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936628 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936640 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-tls-assets\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936648 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-main-db\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936657 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936665 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-config-out\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936673 2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-metrics-client-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936682 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936690 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-secret-alertmanager-main-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936702 2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-cluster-tls-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:51.936850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:51.936731 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f3b0764-5261-4fa2-b9e9-72e299e42dd1-web-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:52.113416 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:52.113373 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:02:52.117769 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:52.117739 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 04:02:52.920058 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:52.919979 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" path="/var/lib/kubelet/pods/8f3b0764-5261-4fa2-b9e9-72e299e42dd1/volumes" Jul 09 04:02:54.719499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719463 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:54.719955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719763 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="prometheus" containerID="cri-o://f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" gracePeriod=600 Jul 09 04:02:54.719955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719798 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy" containerID="cri-o://6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" gracePeriod=600 Jul 09 04:02:54.719955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719841 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-web" containerID="cri-o://6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" gracePeriod=600 Jul 09 04:02:54.719955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719854 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="config-reloader" containerID="cri-o://72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" gracePeriod=600 Jul 09 04:02:54.719955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719910 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" gracePeriod=600 Jul 09 04:02:54.720192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.719862 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="thanos-sidecar" containerID="cri-o://e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" gracePeriod=600 Jul 09 04:02:54.971458 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:54.971401 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.062600 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062567 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062609 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062631 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062670 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062691 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.062774 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062744 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062798 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062838 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062871 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062902 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062947 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062974 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.062999 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063025 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063079 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063104 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.063918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063120 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063182 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063221 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1\") pod \"2122e852-0f7f-44b0-a58d-02b85e66470c\" (UID: \"2122e852-0f7f-44b0-a58d-02b85e66470c\") " Jul 09 04:02:55.063918 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063722 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.064152 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064036 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.064379 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064339 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.064570 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064540 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.064679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.063892 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.064740 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064694 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.064790 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064738 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:55.064975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.064950 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:55.065719 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.065684 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out" (OuterVolumeSpecName: "config-out") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:55.066285 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.066248 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.067051 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067005 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:55.067051 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067038 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config" (OuterVolumeSpecName: "config") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.067201 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067068 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.067335 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067311 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.067444 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067311 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47" (OuterVolumeSpecName: "kube-api-access-lcm47") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "kube-api-access-lcm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:55.067619 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067589 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.067707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.067662 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.068637 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.068606 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.068784 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.068768 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.077294 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.077266 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config" (OuterVolumeSpecName: "web-config") pod "2122e852-0f7f-44b0-a58d-02b85e66470c" (UID: "2122e852-0f7f-44b0-a58d-02b85e66470c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:55.164767 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164728 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-config-out\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164767 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164764 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164778 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164792 2570 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-metrics-client-certs\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164830 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-web-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164842 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164854 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-db\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164866 2570 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164879 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164892 2570 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-kube-rbac-proxy\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164904 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-tls-assets\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164916 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164927 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164941 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164956 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2122e852-0f7f-44b0-a58d-02b85e66470c-configmap-metrics-client-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164970 2570 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-grpc-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.164978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164983 2570 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.165470 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.164997 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2122e852-0f7f-44b0-a58d-02b85e66470c-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.165470 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.165013 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/2122e852-0f7f-44b0-a58d-02b85e66470c-kube-api-access-lcm47\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812398 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" exitCode=0 Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812423 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" exitCode=0 Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812429 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" exitCode=0 Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812434 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" exitCode=0 Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812439 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" exitCode=0 Jul 09 04:02:55.812435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812444 2570 generic.go:356] "Generic (PLEG): container finished" podID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" exitCode=0 Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812501 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812545 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812576 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812622 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812628 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} Jul 09 04:02:55.813178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.812753 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2122e852-0f7f-44b0-a58d-02b85e66470c","Type":"ContainerDied","Data":"ac4e86037c3d85fd6d19816b3c35e4b1954733f46689b873467ba0b35142e046"} Jul 09 04:02:55.821506 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.821479 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.828643 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.828626 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.835369 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.835328 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:55.835773 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.835758 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.839398 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.839375 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:55.842965 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.842939 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.849888 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.849871 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.857252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.857234 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.864802 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.864783 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.864863 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.864789 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:55.865060 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.865040 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.865119 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865071 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.865119 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865096 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865142 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="alertmanager" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865154 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="alertmanager" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865164 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-thanos" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865170 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-thanos" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865176 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" containerName="console" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865181 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" containerName="console" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865187 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="config-reloader" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865195 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="config-reloader" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865209 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy" Jul 09 04:02:55.865215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865215 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865225 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" containerName="console" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865230 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" containerName="console" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865236 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-metric" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865241 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-metric" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865246 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="init-config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865251 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="init-config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865257 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865262 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865271 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="prom-label-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865279 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="prom-label-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865286 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="thanos-sidecar" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865291 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="thanos-sidecar" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865298 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="init-config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865304 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="init-config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865310 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865316 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865321 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865326 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865331 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="prometheus" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865339 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="prometheus" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865374 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865381 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865396 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerName="registry" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865401 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerName="registry" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.865373 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865488 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865480 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865499 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="config-reloader" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865503 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865504 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-thanos" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865577 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865594 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="prometheus" Jul 09 04:02:55.865641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865602 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="thanos-sidecar" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865614 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865627 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="be11c46e-b7a1-4194-b71f-6c59140f8c1d" containerName="registry" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865635 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-metric" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865646 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865656 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b056f69-9a6b-4cdf-915e-bcc0ee781c5a" containerName="console" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865665 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="alertmanager" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865674 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="prom-label-proxy" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865685 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3fa775-c11a-4aed-95bd-c5a6fc516933" containerName="console" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865695 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3b0764-5261-4fa2-b9e9-72e299e42dd1" containerName="kube-rbac-proxy-web" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.865757 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865777 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.865793 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.865994 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866017 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866031 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.866225 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866245 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866258 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.866523 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866547 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.867143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866560 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:02:55.866784 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866802 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.866817 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867016 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867031 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867210 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867223 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867426 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867441 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867651 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867668 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867845 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.867966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.867860 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.868381 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868062 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.868381 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868082 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.868381 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868279 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.868381 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868297 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.868584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868527 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.868584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868543 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.868761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868743 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.868761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868760 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.868959 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868939 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.869030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.868961 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.869206 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869185 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.869249 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869207 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.869522 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869496 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.869587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869522 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.869772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869755 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.869818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869773 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.869979 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869960 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.870021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.869981 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.870187 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870171 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.870239 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870188 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.870441 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870411 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.870441 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870437 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.870659 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870643 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.870701 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870660 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.870878 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870860 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.870929 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.870878 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.871107 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871085 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.871107 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871102 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.871329 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871310 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.871429 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871330 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.871579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871562 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.871650 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871581 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.871756 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871739 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.871819 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871786 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.871819 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.871806 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.872058 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872030 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.872159 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872059 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.872409 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872389 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.872409 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872409 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.872680 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872661 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.872727 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872681 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.872933 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872911 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.872980 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.872936 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.873158 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873128 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.873158 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873149 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.873372 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873331 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.873436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873387 2570 scope.go:117] "RemoveContainer" containerID="6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df" Jul 09 04:02:55.873597 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873579 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df"} err="failed to get container status \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": rpc error: code = NotFound desc = could not find container \"6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df\": container with ID starting with 6f55157271bc5b09560bf2839aab798ed9da14dd7c5672da65e80fa8f608e1df not found: ID does not exist" Jul 09 04:02:55.873640 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873599 2570 scope.go:117] "RemoveContainer" containerID="6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50" Jul 09 04:02:55.873820 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873796 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50"} err="failed to get container status \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": rpc error: code = NotFound desc = could not find container \"6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50\": container with ID starting with 6a739a0ddb703b3e1d061d342f7d1ee3b45bc97a40e3ce3df768f43fb8169c50 not found: ID does not exist" Jul 09 04:02:55.873867 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.873823 2570 scope.go:117] "RemoveContainer" containerID="6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d" Jul 09 04:02:55.874072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874053 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d"} err="failed to get container status \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": rpc error: code = NotFound desc = could not find container \"6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d\": container with ID starting with 6031db66fbcb35e349f7fa49c91bc3375971af35ee6072179d3d422974c6590d not found: ID does not exist" Jul 09 04:02:55.874124 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874073 2570 scope.go:117] "RemoveContainer" containerID="e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca" Jul 09 04:02:55.874316 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874296 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca"} err="failed to get container status \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": rpc error: code = NotFound desc = could not find container \"e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca\": container with ID starting with e957ef15286645dddec1f4f237f746dea6f79b853936466fac3fc7f894274fca not found: ID does not exist" Jul 09 04:02:55.874411 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874320 2570 scope.go:117] "RemoveContainer" containerID="72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c" Jul 09 04:02:55.874580 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874562 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c"} err="failed to get container status \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": rpc error: code = NotFound desc = could not find container \"72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c\": container with ID starting with 72df27cc1bfb5e793916d1e9f5350282cb77dcdea632f32b913ca4017dce5a9c not found: ID does not exist" Jul 09 04:02:55.874629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874580 2570 scope.go:117] "RemoveContainer" containerID="f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726" Jul 09 04:02:55.874814 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874795 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726"} err="failed to get container status \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": rpc error: code = NotFound desc = could not find container \"f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726\": container with ID starting with f0d5fca5d0924d7cbb83666cab902dfc39d3771e03b826128129cbf3a32ef726 not found: ID does not exist" Jul 09 04:02:55.874862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.874814 2570 scope.go:117] "RemoveContainer" containerID="5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a" Jul 09 04:02:55.875035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.875018 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a"} err="failed to get container status \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": rpc error: code = NotFound desc = could not find container \"5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a\": container with ID starting with 5675b9b984f01492eecbe5f64d217aaaf6b2d47462310dd40a20e10b0791b24a not found: ID does not exist" Jul 09 04:02:55.881135 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881118 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 04:02:55.881282 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881271 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 04:02:55.881328 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881299 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 04:02:55.881399 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881375 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 04:02:55.881399 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881345 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 04:02:55.881505 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881491 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 04:02:55.881556 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881526 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 04:02:55.881621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881603 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 04:02:55.881835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881823 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 04:02:55.881876 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881836 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h98tj\"" Jul 09 04:02:55.882006 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.881995 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 04:02:55.882046 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.882020 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 04:02:55.882046 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.882027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 04:02:55.884716 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.884701 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-brsfh10uukuca\"" Jul 09 04:02:55.900991 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.900968 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:55.903117 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.903101 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 04:02:55.939327 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.939295 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 04:02:55.971731 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.971703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.971848 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.971740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.971848 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.971773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.971848 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.971799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-config-out\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.976814 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.971826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.976814 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.972363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.976814 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.972405 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977041 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.976911 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977041 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.976979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977041 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977569 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977636 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-web-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977759 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977730 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977795 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977834 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977878 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:55.977878 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:55.977870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhms9\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-kube-api-access-vhms9\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhms9\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-kube-api-access-vhms9\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-config-out\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.079771 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.079953 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-web-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080713 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.080887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.080879 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.082584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082226 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.082584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082320 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.082584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082466 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.082802 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082660 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.082802 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082738 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.083305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082943 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.083305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.082991 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.083305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.083113 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.083546 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.083422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82150a79-ab36-4b9e-803d-6e2ea4edd657-config-out\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.083604 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.083539 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.084285 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.084263 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.084398 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.084303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.085007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.084987 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.085203 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.085183 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-web-config\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.085933 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.085900 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82150a79-ab36-4b9e-803d-6e2ea4edd657-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.086589 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.086569 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82150a79-ab36-4b9e-803d-6e2ea4edd657-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.087246 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.087216 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhms9\" (UniqueName: \"kubernetes.io/projected/82150a79-ab36-4b9e-803d-6e2ea4edd657-kube-api-access-vhms9\") pod \"prometheus-k8s-0\" (UID: \"82150a79-ab36-4b9e-803d-6e2ea4edd657\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.180758 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.180710 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:02:56.319980 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.319952 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 04:02:56.322472 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:02:56.322443 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82150a79_ab36_4b9e_803d_6e2ea4edd657.slice/crio-54b27e6ad2e19d7051b714a5e295786f37492ed05f23f5674c74ddfb78d45af5 WatchSource:0}: Error finding container 54b27e6ad2e19d7051b714a5e295786f37492ed05f23f5674c74ddfb78d45af5: Status 404 returned error can't find the container with id 54b27e6ad2e19d7051b714a5e295786f37492ed05f23f5674c74ddfb78d45af5 Jul 09 04:02:56.819870 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.819837 2570 generic.go:356] "Generic (PLEG): container finished" podID="82150a79-ab36-4b9e-803d-6e2ea4edd657" containerID="e6ca5dbdd047771f3fdf4b51a2fef4454fbb5396122bc5d4d85df258fd5d0403" exitCode=0 Jul 09 04:02:56.819870 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.819875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerDied","Data":"e6ca5dbdd047771f3fdf4b51a2fef4454fbb5396122bc5d4d85df258fd5d0403"} Jul 09 04:02:56.820457 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.819925 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"54b27e6ad2e19d7051b714a5e295786f37492ed05f23f5674c74ddfb78d45af5"} Jul 09 04:02:56.919212 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:56.919186 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2122e852-0f7f-44b0-a58d-02b85e66470c" path="/var/lib/kubelet/pods/2122e852-0f7f-44b0-a58d-02b85e66470c/volumes" Jul 09 04:02:57.829595 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"548604d95d3908068edcb038b92985c23ea1255b85af0c023ed6202b447bed1e"} Jul 09 04:02:57.830072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"64c107003d1931e59a221d1acc0488e1cabf1acb485692362735eef1fda44f63"} Jul 09 04:02:57.830072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"18db5d40d10137978540caa09f1930dc2f89dfe5bc10a451928573c253fde752"} Jul 09 04:02:57.830072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829671 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"2b40646250e264a7d11077d81ea7dc6c048dc0f04902933e3764ec488fad0df7"} Jul 09 04:02:57.830072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829696 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"e4af35586ac74954f60ef36289f040037a88ca56d2fc5d67124a3333e932bef9"} Jul 09 04:02:57.830072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.829723 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82150a79-ab36-4b9e-803d-6e2ea4edd657","Type":"ContainerStarted","Data":"bf3e685f2ffba44120972d56f5fc8755cf01ca0db145d1bdf416b8a952f1c5e6"} Jul 09 04:02:57.856944 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:02:57.856889 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.837754475 podStartE2EDuration="2.856869241s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.114766ms" isStatefulPod=false podCreationTimestamp="2026-07-09 04:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:57.855403962 +0000 UTC m=+177.459362704" watchObservedRunningTime="2026-07-09 04:02:57.856869241 +0000 UTC m=+177.460827983" Jul 09 04:03:01.181159 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:01.181058 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:03:18.259365 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:18.259318 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:03:43.288201 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.288152 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7fc85c95fb-74r9c" podUID="bdf7fc69-8244-460c-b615-edb157dc7589" containerName="console" containerID="cri-o://56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c" gracePeriod=15 Jul 09 04:03:43.539769 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.539710 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:03:43.662021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.661991 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msgxm\" (UniqueName: \"kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662114 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662136 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662151 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662180 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662171 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662411 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662202 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662411 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662227 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config\") pod \"bdf7fc69-8244-460c-b615-edb157dc7589\" (UID: \"bdf7fc69-8244-460c-b615-edb157dc7589\") " Jul 09 04:03:43.662707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662673 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:43.662843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662745 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca" (OuterVolumeSpecName: "service-ca") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:43.662843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662800 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config" (OuterVolumeSpecName: "console-config") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:43.662953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.662905 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:43.664387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.664338 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:03:43.664387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.664373 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm" (OuterVolumeSpecName: "kube-api-access-msgxm") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "kube-api-access-msgxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:03:43.664563 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.664481 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bdf7fc69-8244-460c-b615-edb157dc7589" (UID: "bdf7fc69-8244-460c-b615-edb157dc7589"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:03:43.763950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763898 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.763950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763944 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-oauth-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.763950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763953 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-trusted-ca-bundle\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.763950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763962 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-serving-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.763950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763971 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf7fc69-8244-460c-b615-edb157dc7589-console-oauth-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.764254 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763978 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf7fc69-8244-460c-b615-edb157dc7589-console-config\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:43.764254 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:43.763987 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msgxm\" (UniqueName: \"kubernetes.io/projected/bdf7fc69-8244-460c-b615-edb157dc7589-kube-api-access-msgxm\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:03:44.012453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.012413 2570 generic.go:356] "Generic (PLEG): container finished" podID="bdf7fc69-8244-460c-b615-edb157dc7589" containerID="56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c" exitCode=0 Jul 09 04:03:44.012627 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.012466 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc85c95fb-74r9c" Jul 09 04:03:44.012627 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.012477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc85c95fb-74r9c" event={"ID":"bdf7fc69-8244-460c-b615-edb157dc7589","Type":"ContainerDied","Data":"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c"} Jul 09 04:03:44.012627 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.012528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc85c95fb-74r9c" event={"ID":"bdf7fc69-8244-460c-b615-edb157dc7589","Type":"ContainerDied","Data":"3a04e0959ce187724e130a25ff7a2404c60498e950d0418d1272a28fbcf2030b"} Jul 09 04:03:44.012627 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.012554 2570 scope.go:117] "RemoveContainer" containerID="56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c" Jul 09 04:03:44.023169 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.023126 2570 scope.go:117] "RemoveContainer" containerID="56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c" Jul 09 04:03:44.023899 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:03:44.023869 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c\": container with ID starting with 56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c not found: ID does not exist" containerID="56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c" Jul 09 04:03:44.024005 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.023910 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c"} err="failed to get container status \"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c\": rpc error: code = NotFound desc = could not find container \"56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c\": container with ID starting with 56bb0002651f48edfade13e617b3532c2da5a283be74b116029bfe2a01446c4c not found: ID does not exist" Jul 09 04:03:44.032545 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.032520 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:03:44.035846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.035824 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fc85c95fb-74r9c"] Jul 09 04:03:44.924060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:44.924026 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf7fc69-8244-460c-b615-edb157dc7589" path="/var/lib/kubelet/pods/bdf7fc69-8244-460c-b615-edb157dc7589/volumes" Jul 09 04:03:56.181530 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:56.181496 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:03:56.197262 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:56.197236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:03:57.081091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:03:57.081057 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 04:04:13.053343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.053308 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ht4xk"] Jul 09 04:04:13.053801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.053646 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf7fc69-8244-460c-b615-edb157dc7589" containerName="console" Jul 09 04:04:13.053801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.053660 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf7fc69-8244-460c-b615-edb157dc7589" containerName="console" Jul 09 04:04:13.053801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.053728 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf7fc69-8244-460c-b615-edb157dc7589" containerName="console" Jul 09 04:04:13.057882 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.057862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.059888 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.059867 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t7rkz\"" Jul 09 04:04:13.060005 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.059867 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 04:04:13.060005 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.059874 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 04:04:13.066699 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.066677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ht4xk"] Jul 09 04:04:13.114641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.114608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg7k\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-kube-api-access-bvg7k\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.114798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.114650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-bound-sa-token\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.215955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.215917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-bound-sa-token\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.216162 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.216004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg7k\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-kube-api-access-bvg7k\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.224124 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.224092 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-bound-sa-token\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.224467 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.224434 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg7k\" (UniqueName: \"kubernetes.io/projected/3a69ecfc-c165-4718-b31c-ab08e4266439-kube-api-access-bvg7k\") pod \"cert-manager-55b4d954d5-ht4xk\" (UID: \"3a69ecfc-c165-4718-b31c-ab08e4266439\") " pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.374446 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.374324 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ht4xk" Jul 09 04:04:13.500485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:13.500362 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ht4xk"] Jul 09 04:04:13.503322 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:04:13.503295 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a69ecfc_c165_4718_b31c_ab08e4266439.slice/crio-537695c979a98c1299cdf9b509be04a176d63427cd63b6bc86b5482aae235b50 WatchSource:0}: Error finding container 537695c979a98c1299cdf9b509be04a176d63427cd63b6bc86b5482aae235b50: Status 404 returned error can't find the container with id 537695c979a98c1299cdf9b509be04a176d63427cd63b6bc86b5482aae235b50 Jul 09 04:04:14.131846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:14.131811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ht4xk" event={"ID":"3a69ecfc-c165-4718-b31c-ab08e4266439","Type":"ContainerStarted","Data":"537695c979a98c1299cdf9b509be04a176d63427cd63b6bc86b5482aae235b50"} Jul 09 04:04:17.148271 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:17.148233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ht4xk" event={"ID":"3a69ecfc-c165-4718-b31c-ab08e4266439","Type":"ContainerStarted","Data":"927060946f0029a26e3b34540febed4f47d782a263515b9d9280ad4eb170373f"} Jul 09 04:04:38.389697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.389578 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ht4xk" podStartSLOduration=21.870739383 podStartE2EDuration="25.389555041s" totalImagesPullingTime="3.518815658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:17.164271496 +0000 UTC m=+256.768230238" watchObservedRunningTime="2026-07-09 04:04:38.389555041 +0000 UTC m=+277.993513783" Jul 09 04:04:38.390852 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.390827 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw"] Jul 09 04:04:38.394192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.394176 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.397882 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.397863 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qqrfg\"" Jul 09 04:04:38.398009 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.397914 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 04:04:38.398076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.398018 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 04:04:38.398076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.398028 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 04:04:38.398076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.398048 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 04:04:38.424230 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.424199 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw"] Jul 09 04:04:38.539275 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.539204 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.539485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.539294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.539485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.539376 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzk9c\" (UniqueName: \"kubernetes.io/projected/01ea5194-86e0-4f90-bbc8-8402e3414c4c-kube-api-access-qzk9c\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.539485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.539424 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.539485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.539443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640243 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640243 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640582 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640582 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzk9c\" (UniqueName: \"kubernetes.io/projected/01ea5194-86e0-4f90-bbc8-8402e3414c4c-kube-api-access-qzk9c\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640582 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.640738 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.640715 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.642508 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.642474 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ea5194-86e0-4f90-bbc8-8402e3414c4c-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.642848 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.642830 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.642888 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.642831 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ea5194-86e0-4f90-bbc8-8402e3414c4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.649281 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.649256 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzk9c\" (UniqueName: \"kubernetes.io/projected/01ea5194-86e0-4f90-bbc8-8402e3414c4c-kube-api-access-qzk9c\") pod \"opendatahub-operator-controller-manager-67dd5f9985-lr7gw\" (UID: \"01ea5194-86e0-4f90-bbc8-8402e3414c4c\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.704975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.704938 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:38.853516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:38.853487 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw"] Jul 09 04:04:38.855895 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:04:38.855866 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ea5194_86e0_4f90_bbc8_8402e3414c4c.slice/crio-95c8f93a980b39950333bbf51bea320043f50c73e999a21d1bf9744f92ca2a9c WatchSource:0}: Error finding container 95c8f93a980b39950333bbf51bea320043f50c73e999a21d1bf9744f92ca2a9c: Status 404 returned error can't find the container with id 95c8f93a980b39950333bbf51bea320043f50c73e999a21d1bf9744f92ca2a9c Jul 09 04:04:39.235873 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:39.235832 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" event={"ID":"01ea5194-86e0-4f90-bbc8-8402e3414c4c","Type":"ContainerStarted","Data":"95c8f93a980b39950333bbf51bea320043f50c73e999a21d1bf9744f92ca2a9c"} Jul 09 04:04:42.251652 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:42.251565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" event={"ID":"01ea5194-86e0-4f90-bbc8-8402e3414c4c","Type":"ContainerStarted","Data":"fe5d6220ec6df7cec5e9f9743dcee9b34dfafa94fc3d1fb4151ac3f4ef787609"} Jul 09 04:04:43.257990 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:43.257953 2570 generic.go:356] "Generic (PLEG): container finished" podID="01ea5194-86e0-4f90-bbc8-8402e3414c4c" containerID="fe5d6220ec6df7cec5e9f9743dcee9b34dfafa94fc3d1fb4151ac3f4ef787609" exitCode=0 Jul 09 04:04:43.258420 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:43.258044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" event={"ID":"01ea5194-86e0-4f90-bbc8-8402e3414c4c","Type":"ContainerDied","Data":"fe5d6220ec6df7cec5e9f9743dcee9b34dfafa94fc3d1fb4151ac3f4ef787609"} Jul 09 04:04:44.264679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:44.264589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" event={"ID":"01ea5194-86e0-4f90-bbc8-8402e3414c4c","Type":"ContainerStarted","Data":"d877a8c49aaa1d85b029fb11d2ef556e512c4609d6e8d1dabb4745baf8fbc065"} Jul 09 04:04:44.265109 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:44.264735 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:44.283979 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:44.283927 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" podStartSLOduration=1.518358742 podStartE2EDuration="6.283910874s" totalImagesPullingTime="3.586917781s" totalInitContainerRuntime="1.178634351s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:44.283339737 +0000 UTC m=+283.887298480" watchObservedRunningTime="2026-07-09 04:04:44.283910874 +0000 UTC m=+283.887869615" Jul 09 04:04:55.271730 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:55.271695 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-lr7gw" Jul 09 04:04:57.998160 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:57.998125 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t"] Jul 09 04:04:58.001339 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.001322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.004740 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.004717 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fchsg\"" Jul 09 04:04:58.005390 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.005342 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 04:04:58.008650 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.008635 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 04:04:58.023076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.023054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t"] Jul 09 04:04:58.101776 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.101736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d3f3d68-b055-4244-a04b-371c52da3e19-tmp\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.101955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.101798 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3f3d68-b055-4244-a04b-371c52da3e19-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.101955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.101821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjmg\" (UniqueName: \"kubernetes.io/projected/4d3f3d68-b055-4244-a04b-371c52da3e19-kube-api-access-2qjmg\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.203021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.202982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3f3d68-b055-4244-a04b-371c52da3e19-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.203021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.203019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjmg\" (UniqueName: \"kubernetes.io/projected/4d3f3d68-b055-4244-a04b-371c52da3e19-kube-api-access-2qjmg\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.203236 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.203085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d3f3d68-b055-4244-a04b-371c52da3e19-tmp\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.205435 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.205412 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d3f3d68-b055-4244-a04b-371c52da3e19-tmp\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.205609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.205592 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3f3d68-b055-4244-a04b-371c52da3e19-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.220741 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.220709 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjmg\" (UniqueName: \"kubernetes.io/projected/4d3f3d68-b055-4244-a04b-371c52da3e19-kube-api-access-2qjmg\") pod \"kube-auth-proxy-7b68b5bb4-nzt2t\" (UID: \"4d3f3d68-b055-4244-a04b-371c52da3e19\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.257854 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.257769 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-txdhh"] Jul 09 04:04:58.261047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.261024 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.263106 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.263072 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 04:04:58.263375 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.263344 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t24cg\"" Jul 09 04:04:58.270191 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.270167 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-txdhh"] Jul 09 04:04:58.303600 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.303568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22tv\" (UniqueName: \"kubernetes.io/projected/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-kube-api-access-q22tv\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.303794 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.303635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.310409 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.310384 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" Jul 09 04:04:58.404986 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.404944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.405160 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.405027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22tv\" (UniqueName: \"kubernetes.io/projected/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-kube-api-access-q22tv\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.405160 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:04:58.405137 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 04:04:58.405270 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:04:58.405221 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert podName:b11e1980-9db0-41c8-b7f9-bbf072fc3fe5 nodeName:}" failed. No retries permitted until 2026-07-09 04:04:58.905203057 +0000 UTC m=+298.509161783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert") pod "odh-model-controller-7998c67dc4-txdhh" (UID: "b11e1980-9db0-41c8-b7f9-bbf072fc3fe5") : secret "odh-model-controller-webhook-cert" not found Jul 09 04:04:58.416067 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.416041 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22tv\" (UniqueName: \"kubernetes.io/projected/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-kube-api-access-q22tv\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.446983 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.446953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t"] Jul 09 04:04:58.449880 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:04:58.449851 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3f3d68_b055_4244_a04b_371c52da3e19.slice/crio-6f5c9f513842af52a2f33636f007136fa116e2330eee30d0715811ba39d3fc46 WatchSource:0}: Error finding container 6f5c9f513842af52a2f33636f007136fa116e2330eee30d0715811ba39d3fc46: Status 404 returned error can't find the container with id 6f5c9f513842af52a2f33636f007136fa116e2330eee30d0715811ba39d3fc46 Jul 09 04:04:58.910807 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.910774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:58.913156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:58.913137 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11e1980-9db0-41c8-b7f9-bbf072fc3fe5-cert\") pod \"odh-model-controller-7998c67dc4-txdhh\" (UID: \"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:59.143752 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.143713 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:04:59.147121 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.147095 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.149497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.149442 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 04:04:59.149636 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.149445 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 04:04:59.149636 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.149584 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 04:04:59.149835 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.149815 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 04:04:59.150026 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.149909 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7wq2m\"" Jul 09 04:04:59.157792 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.157768 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:04:59.171921 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.171840 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:04:59.213765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.213765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.213765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.213765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.213765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.214095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rpf\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.214095 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.213833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314395 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314554 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314554 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314683 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rpf\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.314887 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.314767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.315772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.315743 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.317502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.317427 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.317623 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.317609 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.317791 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.317768 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.317901 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.317875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.329306 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.329251 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.329714 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.329667 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rpf\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf\") pod \"istiod-openshift-gateway-7574459656-tzj2r\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.331447 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.331420 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-txdhh"] Jul 09 04:04:59.338216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.338150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" event={"ID":"4d3f3d68-b055-4244-a04b-371c52da3e19","Type":"ContainerStarted","Data":"6f5c9f513842af52a2f33636f007136fa116e2330eee30d0715811ba39d3fc46"} Jul 09 04:04:59.461219 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.461134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:04:59.627458 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:04:59.627422 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:04:59.630708 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:04:59.630680 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ad37b5_9e4d_4e7d_b535_b57c4cd1ac93.slice/crio-422aef22f19563b90b04c42155fd0d2090bb3185c9d72f09453df926b9c4c4c0 WatchSource:0}: Error finding container 422aef22f19563b90b04c42155fd0d2090bb3185c9d72f09453df926b9c4c4c0: Status 404 returned error can't find the container with id 422aef22f19563b90b04c42155fd0d2090bb3185c9d72f09453df926b9c4c4c0 Jul 09 04:05:00.345957 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:00.345901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" event={"ID":"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93","Type":"ContainerStarted","Data":"422aef22f19563b90b04c42155fd0d2090bb3185c9d72f09453df926b9c4c4c0"} Jul 09 04:05:00.347492 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:00.347440 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" event={"ID":"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5","Type":"ContainerStarted","Data":"2b384a90f3813625dfa20035d41ed3ee8beac2f3606a6f2dfc2b3a573f487732"} Jul 09 04:05:02.982799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:02.982762 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 04:05:03.001029 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.001003 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:05:03.002142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.002116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:05:03.014886 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.014863 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 04:05:03.364668 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.364630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" event={"ID":"4d3f3d68-b055-4244-a04b-371c52da3e19","Type":"ContainerStarted","Data":"a5914b6ca2821def63c975523205e6f73fe7c887101b65d8cf55968a9999ff44"} Jul 09 04:05:03.367021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.366975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" event={"ID":"b11e1980-9db0-41c8-b7f9-bbf072fc3fe5","Type":"ContainerStarted","Data":"94a6f59236c9679b19d6fb398b8fc962dfcb04710e59524e87904056ec5ebf05"} Jul 09 04:05:03.367172 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.367054 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:05:03.381257 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.381203 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-nzt2t" podStartSLOduration=1.852733741 podStartE2EDuration="6.381183069s" totalImagesPullingTime="4.528449328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:03.378961534 +0000 UTC m=+302.982920277" watchObservedRunningTime="2026-07-09 04:05:03.381183069 +0000 UTC m=+302.985141809" Jul 09 04:05:03.398404 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:03.397864 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" podStartSLOduration=1.691071553 podStartE2EDuration="5.397840212s" totalImagesPullingTime="3.706768659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:03.397580226 +0000 UTC m=+303.001539015" watchObservedRunningTime="2026-07-09 04:05:03.397840212 +0000 UTC m=+303.001798953" Jul 09 04:05:04.272186 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:04.272147 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:05:04.272460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:04.272215 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:05:04.375582 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:04.375531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" event={"ID":"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93","Type":"ContainerStarted","Data":"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87"} Jul 09 04:05:04.375794 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:04.375710 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:05:04.397029 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:04.396980 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" podStartSLOduration=0.757769992 podStartE2EDuration="5.396962017s" totalImagesPullingTime="4.639192025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:04.396829096 +0000 UTC m=+304.000787838" watchObservedRunningTime="2026-07-09 04:05:04.396962017 +0000 UTC m=+304.000920757" Jul 09 04:05:05.380894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:05.380867 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:05:08.574013 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.573977 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt"] Jul 09 04:05:08.577846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.577822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.601037 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.601000 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 04:05:08.601235 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.601080 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-n8twl\"" Jul 09 04:05:08.602661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.602633 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj"] Jul 09 04:05:08.606532 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.606507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt"] Jul 09 04:05:08.606651 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.606612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.610544 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.610518 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 04:05:08.610756 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.610738 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cx2cr\"" Jul 09 04:05:08.635419 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.635385 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj"] Jul 09 04:05:08.710191 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.710152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad26ce3b-e76a-4f79-a763-70dfce342f0c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.710395 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.710196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.710395 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.710286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzn85\" (UniqueName: \"kubernetes.io/projected/ad26ce3b-e76a-4f79-a763-70dfce342f0c-kube-api-access-bzn85\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.710477 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.710422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rzq\" (UniqueName: \"kubernetes.io/projected/88f92474-496f-4511-98a9-e6a68bd39469-kube-api-access-h2rzq\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.811007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.810965 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad26ce3b-e76a-4f79-a763-70dfce342f0c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.811231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.811016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.811231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.811054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzn85\" (UniqueName: \"kubernetes.io/projected/ad26ce3b-e76a-4f79-a763-70dfce342f0c-kube-api-access-bzn85\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.811231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.811139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rzq\" (UniqueName: \"kubernetes.io/projected/88f92474-496f-4511-98a9-e6a68bd39469-kube-api-access-h2rzq\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.811231 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:05:08.811188 2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 09 04:05:08.811523 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:05:08.811266 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert podName:88f92474-496f-4511-98a9-e6a68bd39469 nodeName:}" failed. No retries permitted until 2026-07-09 04:05:09.311248609 +0000 UTC m=+308.915207331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert") pod "kserve-controller-manager-d6f4b65b8-7r6lt" (UID: "88f92474-496f-4511-98a9-e6a68bd39469") : secret "kserve-webhook-server-cert" not found Jul 09 04:05:08.813430 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.813407 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad26ce3b-e76a-4f79-a763-70dfce342f0c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.821924 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.821897 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzn85\" (UniqueName: \"kubernetes.io/projected/ad26ce3b-e76a-4f79-a763-70dfce342f0c-kube-api-access-bzn85\") pod \"llmisvc-controller-manager-777c6bd7d5-r24tj\" (UID: \"ad26ce3b-e76a-4f79-a763-70dfce342f0c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:08.824600 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.824539 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rzq\" (UniqueName: \"kubernetes.io/projected/88f92474-496f-4511-98a9-e6a68bd39469-kube-api-access-h2rzq\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:08.916466 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:08.916425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:09.059575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.059543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj"] Jul 09 04:05:09.061695 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:05:09.061659 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podad26ce3b_e76a_4f79_a763_70dfce342f0c.slice/crio-fa295bd9249a27cac2a4ff4b678a09dbb717acf712e6d2217ae5d3d6a5495838 WatchSource:0}: Error finding container fa295bd9249a27cac2a4ff4b678a09dbb717acf712e6d2217ae5d3d6a5495838: Status 404 returned error can't find the container with id fa295bd9249a27cac2a4ff4b678a09dbb717acf712e6d2217ae5d3d6a5495838 Jul 09 04:05:09.063156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.063136 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:05:09.316236 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.316201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:09.318634 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.318607 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f92474-496f-4511-98a9-e6a68bd39469-cert\") pod \"kserve-controller-manager-d6f4b65b8-7r6lt\" (UID: \"88f92474-496f-4511-98a9-e6a68bd39469\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:09.397944 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.397907 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" event={"ID":"ad26ce3b-e76a-4f79-a763-70dfce342f0c","Type":"ContainerStarted","Data":"fa295bd9249a27cac2a4ff4b678a09dbb717acf712e6d2217ae5d3d6a5495838"} Jul 09 04:05:09.488531 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.488485 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:09.643897 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:09.643830 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt"] Jul 09 04:05:09.645968 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:05:09.645934 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f92474_496f_4511_98a9_e6a68bd39469.slice/crio-2faef55d8240bd0884da84313b11084bc349c8f455c7b34ff256cef3b0c1db4d WatchSource:0}: Error finding container 2faef55d8240bd0884da84313b11084bc349c8f455c7b34ff256cef3b0c1db4d: Status 404 returned error can't find the container with id 2faef55d8240bd0884da84313b11084bc349c8f455c7b34ff256cef3b0c1db4d Jul 09 04:05:10.406920 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:10.406847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" event={"ID":"88f92474-496f-4511-98a9-e6a68bd39469","Type":"ContainerStarted","Data":"2faef55d8240bd0884da84313b11084bc349c8f455c7b34ff256cef3b0c1db4d"} Jul 09 04:05:11.468017 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:11.467979 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:05:11.468529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:11.468170 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" podUID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" containerName="discovery" containerID="cri-o://4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87" gracePeriod=30 Jul 09 04:05:12.210860 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.210837 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:05:12.347522 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347480 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347717 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347695 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347744 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347772 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347833 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347910 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.347949 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.347942 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65rpf\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf\") pod \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\" (UID: \"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93\") " Jul 09 04:05:12.348103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.348009 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:05:12.348664 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.348278 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-ca-configmap\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.351089 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.351055 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:05:12.351363 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.351318 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf" (OuterVolumeSpecName: "kube-api-access-65rpf") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "kube-api-access-65rpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:05:12.351516 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.351488 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs" (OuterVolumeSpecName: "local-certs") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:05:12.351654 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.351631 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:05:12.351874 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.351849 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:05:12.352651 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.352623 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts" (OuterVolumeSpecName: "cacerts") pod "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" (UID: "c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:05:12.418919 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.418878 2570 generic.go:356] "Generic (PLEG): container finished" podID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" containerID="4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87" exitCode=0 Jul 09 04:05:12.419105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.418936 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" Jul 09 04:05:12.419105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.418972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" event={"ID":"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93","Type":"ContainerDied","Data":"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87"} Jul 09 04:05:12.419105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.419030 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r" event={"ID":"c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93","Type":"ContainerDied","Data":"422aef22f19563b90b04c42155fd0d2090bb3185c9d72f09453df926b9c4c4c0"} Jul 09 04:05:12.419105 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.419057 2570 scope.go:117] "RemoveContainer" containerID="4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87" Jul 09 04:05:12.449504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449427 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-csr-dns-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.449504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449459 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-cacerts\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.449504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449477 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-local-certs\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.449504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449491 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-kubeconfig\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.449781 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449520 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-istio-token\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.449781 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.449534 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rpf\" (UniqueName: \"kubernetes.io/projected/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93-kube-api-access-65rpf\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:05:12.456445 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.456401 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:05:12.474444 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.474402 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tzj2r"] Jul 09 04:05:12.920865 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.920830 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" path="/var/lib/kubelet/pods/c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93/volumes" Jul 09 04:05:12.987961 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.987762 2570 scope.go:117] "RemoveContainer" containerID="4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87" Jul 09 04:05:12.988170 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:05:12.988152 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87\": container with ID starting with 4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87 not found: ID does not exist" containerID="4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87" Jul 09 04:05:12.988231 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:12.988179 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87"} err="failed to get container status \"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87\": rpc error: code = NotFound desc = could not find container \"4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87\": container with ID starting with 4e70959e2e76944b40a7269c8870a498883600afbac898def910b4a7fd8a2d87 not found: ID does not exist" Jul 09 04:05:13.425409 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.425345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" event={"ID":"88f92474-496f-4511-98a9-e6a68bd39469","Type":"ContainerStarted","Data":"a474c4ca1192511437305fe117ad1ca3fc6ad4e93af4469cd6cb0612485d2e97"} Jul 09 04:05:13.425625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.425481 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:13.427585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.427560 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" event={"ID":"ad26ce3b-e76a-4f79-a763-70dfce342f0c","Type":"ContainerStarted","Data":"ba22a4854b5b8568fcf202fa9cb19c37f8409e8aa90b4f588f86236c0cb8233f"} Jul 09 04:05:13.427740 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.427610 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:13.441133 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.441080 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" podStartSLOduration=2.043881691 podStartE2EDuration="5.441064603s" totalImagesPullingTime="3.397182912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:13.439471715 +0000 UTC m=+313.043430479" watchObservedRunningTime="2026-07-09 04:05:13.441064603 +0000 UTC m=+313.045023345" Jul 09 04:05:13.456006 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:13.455906 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" podStartSLOduration=1.479018545 podStartE2EDuration="5.455888691s" totalImagesPullingTime="3.976870146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:13.455499676 +0000 UTC m=+313.059458416" watchObservedRunningTime="2026-07-09 04:05:13.455888691 +0000 UTC m=+313.059847431" Jul 09 04:05:14.378585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:14.378548 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-txdhh" Jul 09 04:05:44.436925 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:44.436890 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r24tj" Jul 09 04:05:44.440150 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:44.440128 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7r6lt" Jul 09 04:05:58.813525 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.813432 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:05:58.813961 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.813826 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" containerName="discovery" Jul 09 04:05:58.813961 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.813837 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" containerName="discovery" Jul 09 04:05:58.813961 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.813896 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ad37b5-9e4d-4e7d-b535-b57c4cd1ac93" containerName="discovery" Jul 09 04:05:58.816293 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.816275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:05:58.818312 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.818283 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pxjkw\"" Jul 09 04:05:58.818312 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.818291 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 04:05:58.818543 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.818371 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 04:05:58.834544 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.830619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:05:58.953256 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:58.953219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qkg\" (UniqueName: \"kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg\") pod \"kuadrant-operator-catalog-v2b55\" (UID: \"21a07f37-d048-4532-9116-ea182cfa0f95\") " pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:05:59.054729 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.054679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qkg\" (UniqueName: \"kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg\") pod \"kuadrant-operator-catalog-v2b55\" (UID: \"21a07f37-d048-4532-9116-ea182cfa0f95\") " pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:05:59.062276 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.062252 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qkg\" (UniqueName: \"kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg\") pod \"kuadrant-operator-catalog-v2b55\" (UID: \"21a07f37-d048-4532-9116-ea182cfa0f95\") " pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:05:59.135148 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.135056 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:05:59.172310 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.172268 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:05:59.265210 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.265180 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:05:59.267511 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:05:59.267480 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a07f37_d048_4532_9116_ea182cfa0f95.slice/crio-ad92f695e1d40d18ea16f5ce9827e4c600b8a7a10e89a2a38eccd293dec70ed0 WatchSource:0}: Error finding container ad92f695e1d40d18ea16f5ce9827e4c600b8a7a10e89a2a38eccd293dec70ed0: Status 404 returned error can't find the container with id ad92f695e1d40d18ea16f5ce9827e4c600b8a7a10e89a2a38eccd293dec70ed0 Jul 09 04:05:59.387591 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.387505 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hflmz"] Jul 09 04:05:59.390843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.390826 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:05:59.398484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.398407 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hflmz"] Jul 09 04:05:59.457335 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.457295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hddv\" (UniqueName: \"kubernetes.io/projected/22caa450-d143-48fe-92a8-c55b91e3a3e1-kube-api-access-9hddv\") pod \"kuadrant-operator-catalog-hflmz\" (UID: \"22caa450-d143-48fe-92a8-c55b91e3a3e1\") " pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:05:59.558271 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.558227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hddv\" (UniqueName: \"kubernetes.io/projected/22caa450-d143-48fe-92a8-c55b91e3a3e1-kube-api-access-9hddv\") pod \"kuadrant-operator-catalog-hflmz\" (UID: \"22caa450-d143-48fe-92a8-c55b91e3a3e1\") " pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:05:59.565571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.565550 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hddv\" (UniqueName: \"kubernetes.io/projected/22caa450-d143-48fe-92a8-c55b91e3a3e1-kube-api-access-9hddv\") pod \"kuadrant-operator-catalog-hflmz\" (UID: \"22caa450-d143-48fe-92a8-c55b91e3a3e1\") " pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:05:59.649138 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.649042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" event={"ID":"21a07f37-d048-4532-9116-ea182cfa0f95","Type":"ContainerStarted","Data":"ad92f695e1d40d18ea16f5ce9827e4c600b8a7a10e89a2a38eccd293dec70ed0"} Jul 09 04:05:59.701414 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.701374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:05:59.842563 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:05:59.842533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hflmz"] Jul 09 04:05:59.847006 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:05:59.846379 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22caa450_d143_48fe_92a8_c55b91e3a3e1.slice/crio-43530edb712b5a30bc99fce04f70d78a6f72eefb23afa9358fecc92ad1fe72eb WatchSource:0}: Error finding container 43530edb712b5a30bc99fce04f70d78a6f72eefb23afa9358fecc92ad1fe72eb: Status 404 returned error can't find the container with id 43530edb712b5a30bc99fce04f70d78a6f72eefb23afa9358fecc92ad1fe72eb Jul 09 04:06:00.657587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:00.657544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" event={"ID":"22caa450-d143-48fe-92a8-c55b91e3a3e1","Type":"ContainerStarted","Data":"43530edb712b5a30bc99fce04f70d78a6f72eefb23afa9358fecc92ad1fe72eb"} Jul 09 04:06:01.664233 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.664122 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" event={"ID":"21a07f37-d048-4532-9116-ea182cfa0f95","Type":"ContainerStarted","Data":"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3"} Jul 09 04:06:01.664742 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.664239 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" podUID="21a07f37-d048-4532-9116-ea182cfa0f95" containerName="registry-server" containerID="cri-o://28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3" gracePeriod=2 Jul 09 04:06:01.665821 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.665790 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" event={"ID":"22caa450-d143-48fe-92a8-c55b91e3a3e1","Type":"ContainerStarted","Data":"a62c23f418696aabeabcf6684f13fb7ebe5ba1d7450e14a7a6dc673425000761"} Jul 09 04:06:01.679664 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.679606 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" podStartSLOduration=1.5780744420000001 podStartE2EDuration="3.679590231s" totalImagesPullingTime="2.101515789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:01.67765428 +0000 UTC m=+361.281613048" watchObservedRunningTime="2026-07-09 04:06:01.679590231 +0000 UTC m=+361.283548972" Jul 09 04:06:01.690992 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.690945 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" podStartSLOduration=1.1668609349999999 podStartE2EDuration="2.690929099s" totalImagesPullingTime="1.524068164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:01.69054182 +0000 UTC m=+361.294500579" watchObservedRunningTime="2026-07-09 04:06:01.690929099 +0000 UTC m=+361.294887862" Jul 09 04:06:01.933317 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:01.933288 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:06:02.081086 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.081056 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9qkg\" (UniqueName: \"kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg\") pod \"21a07f37-d048-4532-9116-ea182cfa0f95\" (UID: \"21a07f37-d048-4532-9116-ea182cfa0f95\") " Jul 09 04:06:02.083213 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.083177 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg" (OuterVolumeSpecName: "kube-api-access-t9qkg") pod "21a07f37-d048-4532-9116-ea182cfa0f95" (UID: "21a07f37-d048-4532-9116-ea182cfa0f95"). InnerVolumeSpecName "kube-api-access-t9qkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:02.182438 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.182303 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qkg\" (UniqueName: \"kubernetes.io/projected/21a07f37-d048-4532-9116-ea182cfa0f95-kube-api-access-t9qkg\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:06:02.672777 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.672742 2570 generic.go:356] "Generic (PLEG): container finished" podID="21a07f37-d048-4532-9116-ea182cfa0f95" containerID="28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3" exitCode=0 Jul 09 04:06:02.673174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.672801 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" Jul 09 04:06:02.673174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.672819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" event={"ID":"21a07f37-d048-4532-9116-ea182cfa0f95","Type":"ContainerDied","Data":"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3"} Jul 09 04:06:02.673174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.672895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v2b55" event={"ID":"21a07f37-d048-4532-9116-ea182cfa0f95","Type":"ContainerDied","Data":"ad92f695e1d40d18ea16f5ce9827e4c600b8a7a10e89a2a38eccd293dec70ed0"} Jul 09 04:06:02.673174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.672916 2570 scope.go:117] "RemoveContainer" containerID="28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3" Jul 09 04:06:02.682959 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.682945 2570 scope.go:117] "RemoveContainer" containerID="28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3" Jul 09 04:06:02.683245 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:02.683224 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3\": container with ID starting with 28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3 not found: ID does not exist" containerID="28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3" Jul 09 04:06:02.683324 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.683257 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3"} err="failed to get container status \"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3\": rpc error: code = NotFound desc = could not find container \"28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3\": container with ID starting with 28117897f590c8032a5026b0e3bd766345d2b390d8dade053a1833f1e597e8f3 not found: ID does not exist" Jul 09 04:06:02.692655 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.692617 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:06:02.694253 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.694233 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v2b55"] Jul 09 04:06:02.920980 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:02.920945 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a07f37-d048-4532-9116-ea182cfa0f95" path="/var/lib/kubelet/pods/21a07f37-d048-4532-9116-ea182cfa0f95/volumes" Jul 09 04:06:09.701665 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:09.701623 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:06:09.702088 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:09.701689 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:06:09.724914 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:09.724886 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:06:09.746909 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:09.746880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hflmz" Jul 09 04:06:28.921481 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.921450 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt"] Jul 09 04:06:28.921844 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.921781 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a07f37-d048-4532-9116-ea182cfa0f95" containerName="registry-server" Jul 09 04:06:28.921844 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.921791 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a07f37-d048-4532-9116-ea182cfa0f95" containerName="registry-server" Jul 09 04:06:28.921915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.921873 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a07f37-d048-4532-9116-ea182cfa0f95" containerName="registry-server" Jul 09 04:06:28.930032 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.930010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:28.932290 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.932267 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wkj2r\"" Jul 09 04:06:28.940305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:28.940281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt"] Jul 09 04:06:29.025739 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.025703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbwj\" (UniqueName: \"kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.025929 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.025758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.126877 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.126837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.127056 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.126942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbwj\" (UniqueName: \"kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.127276 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.127239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.134823 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.134797 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbwj\" (UniqueName: \"kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.242853 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.242761 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:29.385965 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.385936 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt"] Jul 09 04:06:29.388316 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:29.388290 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a25623_4412_4ee8_843d_8cc5acdf1048.slice/crio-b18561ade3acd71054cc5da30ceaf00952a10dc197310f181a8ac7c967f7e169 WatchSource:0}: Error finding container b18561ade3acd71054cc5da30ceaf00952a10dc197310f181a8ac7c967f7e169: Status 404 returned error can't find the container with id b18561ade3acd71054cc5da30ceaf00952a10dc197310f181a8ac7c967f7e169 Jul 09 04:06:29.799666 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:29.799631 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" event={"ID":"54a25623-4412-4ee8-843d-8cc5acdf1048","Type":"ContainerStarted","Data":"b18561ade3acd71054cc5da30ceaf00952a10dc197310f181a8ac7c967f7e169"} Jul 09 04:06:30.962728 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:30.962692 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn"] Jul 09 04:06:30.966749 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:30.966723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:30.968830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:30.968808 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 04:06:30.968975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:30.968877 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-64r7m\"" Jul 09 04:06:30.975885 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:30.975859 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn"] Jul 09 04:06:31.044018 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.043976 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84tm\" (UniqueName: \"kubernetes.io/projected/f0da4996-4c44-44ca-999d-a53306a2b310-kube-api-access-d84tm\") pod \"dns-operator-controller-manager-864dd78c4c-5p5kn\" (UID: \"f0da4996-4c44-44ca-999d-a53306a2b310\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:31.145143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.145098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84tm\" (UniqueName: \"kubernetes.io/projected/f0da4996-4c44-44ca-999d-a53306a2b310-kube-api-access-d84tm\") pod \"dns-operator-controller-manager-864dd78c4c-5p5kn\" (UID: \"f0da4996-4c44-44ca-999d-a53306a2b310\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:31.157818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.157786 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84tm\" (UniqueName: \"kubernetes.io/projected/f0da4996-4c44-44ca-999d-a53306a2b310-kube-api-access-d84tm\") pod \"dns-operator-controller-manager-864dd78c4c-5p5kn\" (UID: \"f0da4996-4c44-44ca-999d-a53306a2b310\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:31.280601 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.280504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:31.426795 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.426768 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn"] Jul 09 04:06:31.429301 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:31.429267 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0da4996_4c44_44ca_999d_a53306a2b310.slice/crio-819fe1ab628d0190380f97d9abb2f5fa9a31b17ba9619784117718ea34313f0a WatchSource:0}: Error finding container 819fe1ab628d0190380f97d9abb2f5fa9a31b17ba9619784117718ea34313f0a: Status 404 returned error can't find the container with id 819fe1ab628d0190380f97d9abb2f5fa9a31b17ba9619784117718ea34313f0a Jul 09 04:06:31.812228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:31.812182 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" event={"ID":"f0da4996-4c44-44ca-999d-a53306a2b310","Type":"ContainerStarted","Data":"819fe1ab628d0190380f97d9abb2f5fa9a31b17ba9619784117718ea34313f0a"} Jul 09 04:06:33.793826 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.793787 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn"] Jul 09 04:06:33.797621 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.797597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:33.802707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.802683 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8nbxf\"" Jul 09 04:06:33.821842 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.821805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn"] Jul 09 04:06:33.824790 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.824719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" event={"ID":"f0da4996-4c44-44ca-999d-a53306a2b310","Type":"ContainerStarted","Data":"e12356a948c4b6502fcc899757076b1d0c097b7af476323b5e2d294f0aed6033"} Jul 09 04:06:33.824955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.824824 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:33.873116 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.873066 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" podStartSLOduration=1.8919961600000001 podStartE2EDuration="3.873049247s" totalImagesPullingTime="1.981053087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:33.871383958 +0000 UTC m=+393.475342701" watchObservedRunningTime="2026-07-09 04:06:33.873049247 +0000 UTC m=+393.477007991" Jul 09 04:06:33.875717 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.875659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkx92\" (UniqueName: \"kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92\") pod \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" (UID: \"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:33.976663 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.976619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkx92\" (UniqueName: \"kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92\") pod \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" (UID: \"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:33.986030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:33.985996 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkx92\" (UniqueName: \"kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92\") pod \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" (UID: \"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:34.110374 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:34.110269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:34.250104 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:34.249998 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn"] Jul 09 04:06:34.252916 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:34.252872 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fdcec5_aa63_46d9_ad7b_fddaa7d1dabc.slice/crio-7205174c5abe403393a4d782d39f2debbe74941f446443f3f34bf0a9af958a55 WatchSource:0}: Error finding container 7205174c5abe403393a4d782d39f2debbe74941f446443f3f34bf0a9af958a55: Status 404 returned error can't find the container with id 7205174c5abe403393a4d782d39f2debbe74941f446443f3f34bf0a9af958a55 Jul 09 04:06:34.831931 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:34.831770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" event={"ID":"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc","Type":"ContainerStarted","Data":"7205174c5abe403393a4d782d39f2debbe74941f446443f3f34bf0a9af958a55"} Jul 09 04:06:36.845302 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:36.845256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" event={"ID":"54a25623-4412-4ee8-843d-8cc5acdf1048","Type":"ContainerStarted","Data":"59c5a2201a51722c778cb5ea9e61fcd3ea5e0cfcc980cb2103b504d141952666"} Jul 09 04:06:36.845804 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:36.845434 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:36.867201 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:36.867136 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" podStartSLOduration=2.173013308 podStartE2EDuration="8.867114537s" totalImagesPullingTime="6.694101229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:36.864206306 +0000 UTC m=+396.468165072" watchObservedRunningTime="2026-07-09 04:06:36.867114537 +0000 UTC m=+396.471073278" Jul 09 04:06:37.396086 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.396049 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w"] Jul 09 04:06:37.399528 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.399510 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:37.401605 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.401575 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pgjf6\"" Jul 09 04:06:37.420101 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.420056 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w"] Jul 09 04:06:37.509653 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.509607 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z779w\" (UniqueName: \"kubernetes.io/projected/7effc24d-0ee0-4142-9541-7fca9157d238-kube-api-access-z779w\") pod \"authorino-operator-75cc8d5cb7-tbv8w\" (UID: \"7effc24d-0ee0-4142-9541-7fca9157d238\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:37.610655 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.610612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z779w\" (UniqueName: \"kubernetes.io/projected/7effc24d-0ee0-4142-9541-7fca9157d238-kube-api-access-z779w\") pod \"authorino-operator-75cc8d5cb7-tbv8w\" (UID: \"7effc24d-0ee0-4142-9541-7fca9157d238\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:37.622377 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.622325 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z779w\" (UniqueName: \"kubernetes.io/projected/7effc24d-0ee0-4142-9541-7fca9157d238-kube-api-access-z779w\") pod \"authorino-operator-75cc8d5cb7-tbv8w\" (UID: \"7effc24d-0ee0-4142-9541-7fca9157d238\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:37.710199 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.710105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:37.845446 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.845416 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w"] Jul 09 04:06:37.848127 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:37.848090 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7effc24d_0ee0_4142_9541_7fca9157d238.slice/crio-c9ef7569503155cec4b6ca2255bbf210645c66406c3a330c26823a76fd3b5f23 WatchSource:0}: Error finding container c9ef7569503155cec4b6ca2255bbf210645c66406c3a330c26823a76fd3b5f23: Status 404 returned error can't find the container with id c9ef7569503155cec4b6ca2255bbf210645c66406c3a330c26823a76fd3b5f23 Jul 09 04:06:37.852691 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.852652 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" event={"ID":"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc","Type":"ContainerStarted","Data":"04e54f350d9c855adf25fd00e3483dcfb51f7ccbd0142516136eb546e63e44dc"} Jul 09 04:06:37.852793 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.852726 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:37.871104 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:37.871062 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" podStartSLOduration=2.165184666 podStartE2EDuration="4.871046686s" totalImagesPullingTime="2.70586202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:37.869773063 +0000 UTC m=+397.473731821" watchObservedRunningTime="2026-07-09 04:06:37.871046686 +0000 UTC m=+397.475005427" Jul 09 04:06:38.860297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:38.860245 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" event={"ID":"7effc24d-0ee0-4142-9541-7fca9157d238","Type":"ContainerStarted","Data":"c9ef7569503155cec4b6ca2255bbf210645c66406c3a330c26823a76fd3b5f23"} Jul 09 04:06:40.872073 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:40.872038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" event={"ID":"7effc24d-0ee0-4142-9541-7fca9157d238","Type":"ContainerStarted","Data":"24968e5ec210b24ed866dd43c3facae7c2e248eb06e3951e19f23e790946b96a"} Jul 09 04:06:40.872546 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:40.872106 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:40.887683 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:40.887608 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" podStartSLOduration=1.6289255169999999 podStartE2EDuration="3.887593598s" totalImagesPullingTime="2.258668081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:40.886187856 +0000 UTC m=+400.490146622" watchObservedRunningTime="2026-07-09 04:06:40.887593598 +0000 UTC m=+400.491552391" Jul 09 04:06:44.834033 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:44.833994 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5p5kn" Jul 09 04:06:47.855952 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:47.855914 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:48.862928 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:48.862895 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:49.642893 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.642859 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt"] Jul 09 04:06:49.643069 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.643044 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" containerName="manager" containerID="cri-o://59c5a2201a51722c778cb5ea9e61fcd3ea5e0cfcc980cb2103b504d141952666" gracePeriod=2 Jul 09 04:06:49.662060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.662031 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt"] Jul 09 04:06:49.668332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.668307 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:06:49.668796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.668778 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" containerName="manager" Jul 09 04:06:49.668885 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.668801 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" containerName="manager" Jul 09 04:06:49.668938 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.668912 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" containerName="manager" Jul 09 04:06:49.670137 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.670101 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"54a25623-4412-4ee8-843d-8cc5acdf1048\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-46zmt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.672472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.672451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.674190 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.674151 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.681184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.679191 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn"] Jul 09 04:06:49.681184 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.679511 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" containerName="manager" containerID="cri-o://04e54f350d9c855adf25fd00e3483dcfb51f7ccbd0142516136eb546e63e44dc" gracePeriod=2 Jul 09 04:06:49.692530 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.692495 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:06:49.694904 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.694878 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn"] Jul 09 04:06:49.704409 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.704372 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.705257 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.705226 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw"] Jul 09 04:06:49.705907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.705843 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" containerName="manager" Jul 09 04:06:49.705907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.705885 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" containerName="manager" Jul 09 04:06:49.706137 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.705980 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:49.706137 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.706031 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" containerName="manager" Jul 09 04:06:49.709477 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.709456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:49.711232 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.711202 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.721067 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.720029 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw"] Jul 09 04:06:49.734872 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.734837 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:49.825572 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.825523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9w4t\" (UniqueName: \"kubernetes.io/projected/f35e121d-d88e-4d7f-97f2-013ac20547d3-kube-api-access-t9w4t\") pod \"limitador-operator-controller-manager-859d7b78bd-gsshw\" (UID: \"f35e121d-d88e-4d7f-97f2-013ac20547d3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:49.825744 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.825703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.825744 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.825732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226xt\" (UniqueName: \"kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.918919 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.918890 2570 generic.go:356] "Generic (PLEG): container finished" podID="54a25623-4412-4ee8-843d-8cc5acdf1048" containerID="59c5a2201a51722c778cb5ea9e61fcd3ea5e0cfcc980cb2103b504d141952666" exitCode=0 Jul 09 04:06:49.919330 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.918968 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b18561ade3acd71054cc5da30ceaf00952a10dc197310f181a8ac7c967f7e169" Jul 09 04:06:49.920919 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.920890 2570 generic.go:356] "Generic (PLEG): container finished" podID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" containerID="04e54f350d9c855adf25fd00e3483dcfb51f7ccbd0142516136eb546e63e44dc" exitCode=0 Jul 09 04:06:49.926938 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.926915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.927044 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.926945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226xt\" (UniqueName: \"kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.927044 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.926984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9w4t\" (UniqueName: \"kubernetes.io/projected/f35e121d-d88e-4d7f-97f2-013ac20547d3-kube-api-access-t9w4t\") pod \"limitador-operator-controller-manager-859d7b78bd-gsshw\" (UID: \"f35e121d-d88e-4d7f-97f2-013ac20547d3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:49.927496 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.927337 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.937305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.937279 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.938671 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.938650 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9w4t\" (UniqueName: \"kubernetes.io/projected/f35e121d-d88e-4d7f-97f2-013ac20547d3-kube-api-access-t9w4t\") pod \"limitador-operator-controller-manager-859d7b78bd-gsshw\" (UID: \"f35e121d-d88e-4d7f-97f2-013ac20547d3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:49.938829 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.938804 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226xt\" (UniqueName: \"kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whqfq\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:49.939017 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.938996 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.940475 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.940453 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:49.941005 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:49.940991 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:49.942502 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.942482 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:49.943848 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:49.943825 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:50.028028 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.027994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume\") pod \"54a25623-4412-4ee8-843d-8cc5acdf1048\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " Jul 09 04:06:50.028203 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.028050 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnbwj\" (UniqueName: \"kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj\") pod \"54a25623-4412-4ee8-843d-8cc5acdf1048\" (UID: \"54a25623-4412-4ee8-843d-8cc5acdf1048\") " Jul 09 04:06:50.028203 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.028067 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkx92\" (UniqueName: \"kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92\") pod \"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc\" (UID: \"73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc\") " Jul 09 04:06:50.028578 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.028551 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "54a25623-4412-4ee8-843d-8cc5acdf1048" (UID: "54a25623-4412-4ee8-843d-8cc5acdf1048"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:50.030092 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.030071 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92" (OuterVolumeSpecName: "kube-api-access-wkx92") pod "73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" (UID: "73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc"). InnerVolumeSpecName "kube-api-access-wkx92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:50.030149 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.030100 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj" (OuterVolumeSpecName: "kube-api-access-tnbwj") pod "54a25623-4412-4ee8-843d-8cc5acdf1048" (UID: "54a25623-4412-4ee8-843d-8cc5acdf1048"). InnerVolumeSpecName "kube-api-access-tnbwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:50.082085 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.082045 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:50.088135 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.088116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:50.129712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.129677 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a25623-4412-4ee8-843d-8cc5acdf1048-extensions-socket-volume\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:06:50.129712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.129708 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbwj\" (UniqueName: \"kubernetes.io/projected/54a25623-4412-4ee8-843d-8cc5acdf1048-kube-api-access-tnbwj\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:06:50.129712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.129718 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkx92\" (UniqueName: \"kubernetes.io/projected/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc-kube-api-access-wkx92\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:06:50.231973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.231943 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:06:50.235203 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:50.235160 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a85cca_cf3f_43da_97c5_8e337e787c31.slice/crio-a9f2a70df4147388653a6e7f5f748d66ac795c33e88967a3aebfcd8b734ce390 WatchSource:0}: Error finding container a9f2a70df4147388653a6e7f5f748d66ac795c33e88967a3aebfcd8b734ce390: Status 404 returned error can't find the container with id a9f2a70df4147388653a6e7f5f748d66ac795c33e88967a3aebfcd8b734ce390 Jul 09 04:06:50.249064 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.249043 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw"] Jul 09 04:06:50.253570 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:50.253549 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35e121d_d88e_4d7f_97f2_013ac20547d3.slice/crio-7529e0916d4d7f16088990258dbbadc00158fac10f46a879a886fe774c6fd4db WatchSource:0}: Error finding container 7529e0916d4d7f16088990258dbbadc00158fac10f46a879a886fe774c6fd4db: Status 404 returned error can't find the container with id 7529e0916d4d7f16088990258dbbadc00158fac10f46a879a886fe774c6fd4db Jul 09 04:06:50.686084 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.686047 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:06:50.689521 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.689501 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.691396 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:50.691345 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:50.703034 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.703006 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:06:50.725484 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:50.725440 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:50.836769 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.836724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.836962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.836833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbt4\" (UniqueName: \"kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.918968 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:50.918872 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-46zmt\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:50.920477 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:06:50.920452 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpkpn\" is forbidden: User \"system:node:ip-10-0-137-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-40.ec2.internal' and this object" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:50.921200 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.921179 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54a25623-4412-4ee8-843d-8cc5acdf1048" path="/var/lib/kubelet/pods/54a25623-4412-4ee8-843d-8cc5acdf1048/volumes" Jul 09 04:06:50.921571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.921554 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc" path="/var/lib/kubelet/pods/73fdcec5-aa63-46d9-ad7b-fddaa7d1dabc/volumes" Jul 09 04:06:50.928297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.928272 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpkpn" Jul 09 04:06:50.928468 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.928315 2570 scope.go:117] "RemoveContainer" containerID="04e54f350d9c855adf25fd00e3483dcfb51f7ccbd0142516136eb546e63e44dc" Jul 09 04:06:50.930555 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.930514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" event={"ID":"f35e121d-d88e-4d7f-97f2-013ac20547d3","Type":"ContainerStarted","Data":"251f6d396a254aa57b56ad0ac0d9a6ca010c2562361198dff90fd9e43f20a923"} Jul 09 04:06:50.930681 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.930572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" event={"ID":"f35e121d-d88e-4d7f-97f2-013ac20547d3","Type":"ContainerStarted","Data":"7529e0916d4d7f16088990258dbbadc00158fac10f46a879a886fe774c6fd4db"} Jul 09 04:06:50.930829 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.930745 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:06:50.932594 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.932573 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-46zmt" Jul 09 04:06:50.932690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.932603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" event={"ID":"05a85cca-cf3f-43da-97c5-8e337e787c31","Type":"ContainerStarted","Data":"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02"} Jul 09 04:06:50.932690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.932650 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" event={"ID":"05a85cca-cf3f-43da-97c5-8e337e787c31","Type":"ContainerStarted","Data":"a9f2a70df4147388653a6e7f5f748d66ac795c33e88967a3aebfcd8b734ce390"} Jul 09 04:06:50.932773 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.932697 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:06:50.937371 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.937326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbt4\" (UniqueName: \"kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.937475 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.937450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.937787 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.937770 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.944728 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.944706 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbt4\" (UniqueName: \"kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4\") pod \"kuadrant-operator-controller-manager-86878c769f-g9rsh\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:50.958548 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.958502 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" podStartSLOduration=1.958486492 podStartE2EDuration="1.958486492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:50.956490876 +0000 UTC m=+410.560449629" watchObservedRunningTime="2026-07-09 04:06:50.958486492 +0000 UTC m=+410.562445212" Jul 09 04:06:50.978035 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.977967 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" podStartSLOduration=1.9779459419999998 podStartE2EDuration="1.977945942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:50.976618393 +0000 UTC m=+410.580577148" watchObservedRunningTime="2026-07-09 04:06:50.977945942 +0000 UTC m=+410.581904683" Jul 09 04:06:50.999790 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:50.999765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:51.142206 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:51.142182 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:06:51.145129 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:06:51.145099 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad46da13_5e1b_408d_8e2d_91a89eaa4936.slice/crio-63d33cc2b122499f9dc713c6056650a0a6abb8235b58fb0a7b4487f76a42ef5c WatchSource:0}: Error finding container 63d33cc2b122499f9dc713c6056650a0a6abb8235b58fb0a7b4487f76a42ef5c: Status 404 returned error can't find the container with id 63d33cc2b122499f9dc713c6056650a0a6abb8235b58fb0a7b4487f76a42ef5c Jul 09 04:06:51.879871 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:51.879836 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tbv8w" Jul 09 04:06:51.940770 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:51.940733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" event={"ID":"ad46da13-5e1b-408d-8e2d-91a89eaa4936","Type":"ContainerStarted","Data":"181fda6be879231a2690bc152f63b713a3dde2a3b09eb9572a6339a3ec110210"} Jul 09 04:06:51.940770 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:51.940785 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" event={"ID":"ad46da13-5e1b-408d-8e2d-91a89eaa4936","Type":"ContainerStarted","Data":"63d33cc2b122499f9dc713c6056650a0a6abb8235b58fb0a7b4487f76a42ef5c"} Jul 09 04:06:51.957147 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:51.957096 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" podStartSLOduration=1.9570747370000001 podStartE2EDuration="1.957074737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:51.956377135 +0000 UTC m=+411.560335887" watchObservedRunningTime="2026-07-09 04:06:51.957074737 +0000 UTC m=+411.561033479" Jul 09 04:06:52.946907 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:52.946869 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:06:59.925183 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.925141 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84dd999c45-knggw"] Jul 09 04:06:59.929167 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.929143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:06:59.931504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.931469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 04:06:59.931801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.931778 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 04:06:59.931915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.931869 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 04:06:59.932132 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.932109 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 04:06:59.932267 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.932135 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 04:06:59.932267 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.932112 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd6vl\"" Jul 09 04:06:59.932267 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.932118 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 04:06:59.932267 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.932121 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 04:06:59.938409 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.938390 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 04:06:59.942204 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:06:59.942179 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dd999c45-knggw"] Jul 09 04:07:00.016047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-oauth-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfqx\" (UniqueName: \"kubernetes.io/projected/56208214-d183-45e8-bb36-b42398fbbd43-kube-api-access-hgfqx\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-oauth-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016214 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-service-ca\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016403 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016241 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-console-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016403 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-trusted-ca-bundle\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.016403 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.016390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117081 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-oauth-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117081 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfqx\" (UniqueName: \"kubernetes.io/projected/56208214-d183-45e8-bb36-b42398fbbd43-kube-api-access-hgfqx\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-oauth-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-service-ca\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117223 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-console-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-trusted-ca-bundle\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.117955 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117925 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-console-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.118086 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117931 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-oauth-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.118086 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.117963 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-service-ca\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.118476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.118457 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56208214-d183-45e8-bb36-b42398fbbd43-trusted-ca-bundle\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.119690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.119671 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-oauth-config\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.119792 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.119775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56208214-d183-45e8-bb36-b42398fbbd43-console-serving-cert\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.133305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.133276 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfqx\" (UniqueName: \"kubernetes.io/projected/56208214-d183-45e8-bb36-b42398fbbd43-kube-api-access-hgfqx\") pod \"console-84dd999c45-knggw\" (UID: \"56208214-d183-45e8-bb36-b42398fbbd43\") " pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.242569 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.242469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:00.389061 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.389030 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dd999c45-knggw"] Jul 09 04:07:00.390936 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:07:00.390910 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56208214_d183_45e8_bb36_b42398fbbd43.slice/crio-5253b742708f1256901b0bf80776d528554ada2fcce4c51c4137ffcbcf0b22f4 WatchSource:0}: Error finding container 5253b742708f1256901b0bf80776d528554ada2fcce4c51c4137ffcbcf0b22f4: Status 404 returned error can't find the container with id 5253b742708f1256901b0bf80776d528554ada2fcce4c51c4137ffcbcf0b22f4 Jul 09 04:07:00.989725 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.989686 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dd999c45-knggw" event={"ID":"56208214-d183-45e8-bb36-b42398fbbd43","Type":"ContainerStarted","Data":"b0925f85defb8681225dd5a171a39e3fe4838898662277d0970180eb8d440be9"} Jul 09 04:07:00.990291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:00.989752 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dd999c45-knggw" event={"ID":"56208214-d183-45e8-bb36-b42398fbbd43","Type":"ContainerStarted","Data":"5253b742708f1256901b0bf80776d528554ada2fcce4c51c4137ffcbcf0b22f4"} Jul 09 04:07:01.009883 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:01.009831 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84dd999c45-knggw" podStartSLOduration=2.009814785 podStartE2EDuration="2.009814785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:01.008999043 +0000 UTC m=+420.612957798" watchObservedRunningTime="2026-07-09 04:07:01.009814785 +0000 UTC m=+420.613773525" Jul 09 04:07:01.943044 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:01.943005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:07:01.943242 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:01.943150 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gsshw" Jul 09 04:07:03.954096 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:03.954064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:07:04.010706 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.010678 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:07:04.010890 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.010845 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" podUID="05a85cca-cf3f-43da-97c5-8e337e787c31" containerName="manager" containerID="cri-o://3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02" gracePeriod=10 Jul 09 04:07:04.263389 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.263343 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:07:04.457608 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.457574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-226xt\" (UniqueName: \"kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt\") pod \"05a85cca-cf3f-43da-97c5-8e337e787c31\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " Jul 09 04:07:04.457798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.457654 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume\") pod \"05a85cca-cf3f-43da-97c5-8e337e787c31\" (UID: \"05a85cca-cf3f-43da-97c5-8e337e787c31\") " Jul 09 04:07:04.457984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.457957 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "05a85cca-cf3f-43da-97c5-8e337e787c31" (UID: "05a85cca-cf3f-43da-97c5-8e337e787c31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:07:04.459727 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.459667 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt" (OuterVolumeSpecName: "kube-api-access-226xt") pod "05a85cca-cf3f-43da-97c5-8e337e787c31" (UID: "05a85cca-cf3f-43da-97c5-8e337e787c31"). InnerVolumeSpecName "kube-api-access-226xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:04.558972 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.558936 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226xt\" (UniqueName: \"kubernetes.io/projected/05a85cca-cf3f-43da-97c5-8e337e787c31-kube-api-access-226xt\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:07:04.558972 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:04.558966 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05a85cca-cf3f-43da-97c5-8e337e787c31-extensions-socket-volume\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:07:05.021687 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.021648 2570 generic.go:356] "Generic (PLEG): container finished" podID="05a85cca-cf3f-43da-97c5-8e337e787c31" containerID="3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02" exitCode=0 Jul 09 04:07:05.022078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.021704 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" Jul 09 04:07:05.022078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.021730 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" event={"ID":"05a85cca-cf3f-43da-97c5-8e337e787c31","Type":"ContainerDied","Data":"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02"} Jul 09 04:07:05.022078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.021802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq" event={"ID":"05a85cca-cf3f-43da-97c5-8e337e787c31","Type":"ContainerDied","Data":"a9f2a70df4147388653a6e7f5f748d66ac795c33e88967a3aebfcd8b734ce390"} Jul 09 04:07:05.022078 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.021824 2570 scope.go:117] "RemoveContainer" containerID="3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02" Jul 09 04:07:05.030923 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.030903 2570 scope.go:117] "RemoveContainer" containerID="3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02" Jul 09 04:07:05.031225 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:07:05.031198 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02\": container with ID starting with 3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02 not found: ID does not exist" containerID="3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02" Jul 09 04:07:05.031275 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.031234 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02"} err="failed to get container status \"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02\": rpc error: code = NotFound desc = could not find container \"3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02\": container with ID starting with 3921b19be02b7bf9484be1397aeb2dab896067359e58b14b0995cf0eb8abad02 not found: ID does not exist" Jul 09 04:07:05.038056 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.038030 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:07:05.045383 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:05.045360 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whqfq"] Jul 09 04:07:06.920522 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:06.920485 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a85cca-cf3f-43da-97c5-8e337e787c31" path="/var/lib/kubelet/pods/05a85cca-cf3f-43da-97c5-8e337e787c31/volumes" Jul 09 04:07:10.242967 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:10.242937 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:10.243504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:10.242998 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:10.247670 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:10.247647 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:11.057123 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:11.057092 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84dd999c45-knggw" Jul 09 04:07:20.348343 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.348304 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs"] Jul 09 04:07:20.348941 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.348918 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a85cca-cf3f-43da-97c5-8e337e787c31" containerName="manager" Jul 09 04:07:20.349048 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.348943 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a85cca-cf3f-43da-97c5-8e337e787c31" containerName="manager" Jul 09 04:07:20.349127 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.349106 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a85cca-cf3f-43da-97c5-8e337e787c31" containerName="manager" Jul 09 04:07:20.354104 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.354081 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.356317 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.356293 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 04:07:20.356472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.356297 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7gkn8\"" Jul 09 04:07:20.365798 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.365771 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs"] Jul 09 04:07:20.404566 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404566 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404615 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404687 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.404830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404820 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtht\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-kube-api-access-jdtht\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.405063 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.405063 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.404910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505547 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505547 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtht\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-kube-api-access-jdtht\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.505851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.505826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.506218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.506018 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.506218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.506105 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.506218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.506195 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.506406 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.506384 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.506623 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.506603 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.508251 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.508228 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.508436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.508421 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.514496 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.514473 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.522726 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.522700 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtht\" (UniqueName: \"kubernetes.io/projected/a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c-kube-api-access-jdtht\") pod \"maas-default-gateway-openshift-default-74949dcd89-tmvfs\" (UID: \"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.665292 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.665199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:20.831127 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:20.831101 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs"] Jul 09 04:07:20.832937 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:07:20.832908 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda607c53e_79b0_4406_b0f6_c7bc9a7ddc2c.slice/crio-a5749a03bf045afbaa35aa74904466cbbdbf5acdb108f3803d43329475733ec1 WatchSource:0}: Error finding container a5749a03bf045afbaa35aa74904466cbbdbf5acdb108f3803d43329475733ec1: Status 404 returned error can't find the container with id a5749a03bf045afbaa35aa74904466cbbdbf5acdb108f3803d43329475733ec1 Jul 09 04:07:21.104372 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:21.104309 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" event={"ID":"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c","Type":"ContainerStarted","Data":"a5749a03bf045afbaa35aa74904466cbbdbf5acdb108f3803d43329475733ec1"} Jul 09 04:07:23.407953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:23.407915 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:07:23.408207 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:23.407990 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:07:23.408207 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:23.408019 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:07:24.122469 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:24.122416 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" event={"ID":"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c","Type":"ContainerStarted","Data":"481c4556d3e3332fcb9e1b355612d9ee29bf805fdd852c5b749185ac24a57e6f"} Jul 09 04:07:24.145740 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:24.145688 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" podStartSLOduration=1.5730000849999999 podStartE2EDuration="4.145670649s" totalImagesPullingTime="2.572670564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:24.14380398 +0000 UTC m=+443.747762722" watchObservedRunningTime="2026-07-09 04:07:24.145670649 +0000 UTC m=+443.749629394" Jul 09 04:07:24.665712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:24.665670 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:24.670970 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:24.670941 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:25.128102 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.128064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:25.129143 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.129124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:07:25.326604 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.326568 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:25.329521 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.329502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.332440 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.332412 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 04:07:25.333234 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.333212 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ltzz2\"" Jul 09 04:07:25.342057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.342027 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:25.425529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.425438 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:25.452609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.452569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6nw\" (UniqueName: \"kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.452609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.452610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.553883 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.553845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6nw\" (UniqueName: \"kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.553883 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.553890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.554562 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.554543 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.562072 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.562045 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6nw\" (UniqueName: \"kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw\") pod \"limitador-limitador-64c94fd64f-x8tbt\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.642438 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.642397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:25.784265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:25.784245 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:25.786645 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:07:25.786616 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a52a1_9d55_4925_afc0_1f6f873ba056.slice/crio-36dffd917182c45203ae274d623fe2b7037eb27f43433694d4cba63a0c64a8c0 WatchSource:0}: Error finding container 36dffd917182c45203ae274d623fe2b7037eb27f43433694d4cba63a0c64a8c0: Status 404 returned error can't find the container with id 36dffd917182c45203ae274d623fe2b7037eb27f43433694d4cba63a0c64a8c0 Jul 09 04:07:26.134834 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:26.134798 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" event={"ID":"d75a52a1-9d55-4925-afc0-1f6f873ba056","Type":"ContainerStarted","Data":"36dffd917182c45203ae274d623fe2b7037eb27f43433694d4cba63a0c64a8c0"} Jul 09 04:07:29.154871 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:29.154837 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" event={"ID":"d75a52a1-9d55-4925-afc0-1f6f873ba056","Type":"ContainerStarted","Data":"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c"} Jul 09 04:07:29.155282 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:29.154952 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:29.189014 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:29.188960 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" podStartSLOduration=1.573984486 podStartE2EDuration="4.18894208s" totalImagesPullingTime="2.614957594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:29.188196429 +0000 UTC m=+448.792155169" watchObservedRunningTime="2026-07-09 04:07:29.18894208 +0000 UTC m=+448.792900838" Jul 09 04:07:40.161094 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:40.161054 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:44.418517 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:44.418478 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:44.418975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:44.418694 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" podUID="d75a52a1-9d55-4925-afc0-1f6f873ba056" containerName="limitador" containerID="cri-o://421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c" gracePeriod=30 Jul 09 04:07:44.975911 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:44.975883 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:45.144818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.144779 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz6nw\" (UniqueName: \"kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw\") pod \"d75a52a1-9d55-4925-afc0-1f6f873ba056\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " Jul 09 04:07:45.145003 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.144933 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file\") pod \"d75a52a1-9d55-4925-afc0-1f6f873ba056\" (UID: \"d75a52a1-9d55-4925-afc0-1f6f873ba056\") " Jul 09 04:07:45.145305 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.145272 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file" (OuterVolumeSpecName: "config-file") pod "d75a52a1-9d55-4925-afc0-1f6f873ba056" (UID: "d75a52a1-9d55-4925-afc0-1f6f873ba056"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:07:45.146912 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.146886 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw" (OuterVolumeSpecName: "kube-api-access-rz6nw") pod "d75a52a1-9d55-4925-afc0-1f6f873ba056" (UID: "d75a52a1-9d55-4925-afc0-1f6f873ba056"). InnerVolumeSpecName "kube-api-access-rz6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:45.245973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.245944 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a52a1-9d55-4925-afc0-1f6f873ba056-config-file\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:07:45.245973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.245968 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6nw\" (UniqueName: \"kubernetes.io/projected/d75a52a1-9d55-4925-afc0-1f6f873ba056-kube-api-access-rz6nw\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:07:45.247163 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.247137 2570 generic.go:356] "Generic (PLEG): container finished" podID="d75a52a1-9d55-4925-afc0-1f6f873ba056" containerID="421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c" exitCode=0 Jul 09 04:07:45.247308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.247182 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" Jul 09 04:07:45.247308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.247227 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" event={"ID":"d75a52a1-9d55-4925-afc0-1f6f873ba056","Type":"ContainerDied","Data":"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c"} Jul 09 04:07:45.247308 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.247294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x8tbt" event={"ID":"d75a52a1-9d55-4925-afc0-1f6f873ba056","Type":"ContainerDied","Data":"36dffd917182c45203ae274d623fe2b7037eb27f43433694d4cba63a0c64a8c0"} Jul 09 04:07:45.247500 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.247325 2570 scope.go:117] "RemoveContainer" containerID="421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c" Jul 09 04:07:45.257475 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.257457 2570 scope.go:117] "RemoveContainer" containerID="421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c" Jul 09 04:07:45.257763 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:07:45.257744 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c\": container with ID starting with 421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c not found: ID does not exist" containerID="421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c" Jul 09 04:07:45.257812 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.257772 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c"} err="failed to get container status \"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c\": rpc error: code = NotFound desc = could not find container \"421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c\": container with ID starting with 421a67891015b34ac435d248f226e6d89c1ce11296bc54436894ddc0cba5105c not found: ID does not exist" Jul 09 04:07:45.268291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.268266 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:45.270045 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:45.270027 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x8tbt"] Jul 09 04:07:46.924862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.924827 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75a52a1-9d55-4925-afc0-1f6f873ba056" path="/var/lib/kubelet/pods/d75a52a1-9d55-4925-afc0-1f6f873ba056/volumes" Jul 09 04:07:46.962882 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.962851 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-msxqs"] Jul 09 04:07:46.963538 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.963520 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75a52a1-9d55-4925-afc0-1f6f873ba056" containerName="limitador" Jul 09 04:07:46.963589 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.963542 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75a52a1-9d55-4925-afc0-1f6f873ba056" containerName="limitador" Jul 09 04:07:46.963657 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.963646 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75a52a1-9d55-4925-afc0-1f6f873ba056" containerName="limitador" Jul 09 04:07:46.967253 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.967230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:46.969254 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.969230 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 04:07:46.969430 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.969274 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:07:46.969743 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.969720 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-r2hd8\"" Jul 09 04:07:46.969743 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.969744 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:07:46.973007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:46.972986 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-msxqs"] Jul 09 04:07:47.063502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.063462 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcadf-a964-47c8-bb48-9779f902408f-data\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.063502 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.063507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrnt\" (UniqueName: \"kubernetes.io/projected/7b4bcadf-a964-47c8-bb48-9779f902408f-kube-api-access-fwrnt\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.164319 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.164284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcadf-a964-47c8-bb48-9779f902408f-data\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.164537 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.164326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrnt\" (UniqueName: \"kubernetes.io/projected/7b4bcadf-a964-47c8-bb48-9779f902408f-kube-api-access-fwrnt\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.164701 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.164670 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcadf-a964-47c8-bb48-9779f902408f-data\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.174015 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.173986 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrnt\" (UniqueName: \"kubernetes.io/projected/7b4bcadf-a964-47c8-bb48-9779f902408f-kube-api-access-fwrnt\") pod \"postgres-69fb4594cf-msxqs\" (UID: \"7b4bcadf-a964-47c8-bb48-9779f902408f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.281873 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.281779 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:47.421229 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:47.421202 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-msxqs"] Jul 09 04:07:47.423064 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:07:47.423037 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4bcadf_a964_47c8_bb48_9779f902408f.slice/crio-c2f890d085d71f7be5168d83594cc5ff223cd2dd949d9b088cce949a812a9031 WatchSource:0}: Error finding container c2f890d085d71f7be5168d83594cc5ff223cd2dd949d9b088cce949a812a9031: Status 404 returned error can't find the container with id c2f890d085d71f7be5168d83594cc5ff223cd2dd949d9b088cce949a812a9031 Jul 09 04:07:48.268372 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:48.268198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" event={"ID":"7b4bcadf-a964-47c8-bb48-9779f902408f","Type":"ContainerStarted","Data":"c2f890d085d71f7be5168d83594cc5ff223cd2dd949d9b088cce949a812a9031"} Jul 09 04:07:52.292277 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:52.292238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" event={"ID":"7b4bcadf-a964-47c8-bb48-9779f902408f","Type":"ContainerStarted","Data":"001375b43ecaa217fb2515a9b48239437edb8bab18495c5c81ea8e163d16f2f8"} Jul 09 04:07:52.292693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:52.292314 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:07:52.307316 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:52.307269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" podStartSLOduration=1.7023162429999998 podStartE2EDuration="6.307253721s" totalImagesPullingTime="4.604937478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:52.305753578 +0000 UTC m=+471.909712319" watchObservedRunningTime="2026-07-09 04:07:52.307253721 +0000 UTC m=+471.911212567" Jul 09 04:07:58.328181 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:07:58.328150 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-msxqs" Jul 09 04:08:04.936134 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.936098 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:04.944434 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.944411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:04.946801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.946734 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 04:08:04.946801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.946763 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7z8mm\"" Jul 09 04:08:04.947080 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.946766 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 04:08:04.947753 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:04.947728 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:05.035312 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.035270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndsmk\" (UniqueName: \"kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.035551 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.035318 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.106520 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.106477 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:08:05.109234 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.109208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.117645 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.117620 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:08:05.136288 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.136247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndsmk\" (UniqueName: \"kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.136499 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.136291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.138926 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.138896 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.145478 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.145448 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndsmk\" (UniqueName: \"kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk\") pod \"maas-controller-54c555c58c-424lj\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.237829 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.237734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.237829 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.237795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6h9\" (UniqueName: \"kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.257666 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.257636 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:05.338957 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.338921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.339142 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.338978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6h9\" (UniqueName: \"kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.341273 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.341248 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.347500 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.347475 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6h9\" (UniqueName: \"kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9\") pod \"maas-controller-6cbfc7556c-kvgrj\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.400061 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.400034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:05.402033 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:08:05.402007 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4498532_947d_4d29_99b3_b2f0605f71fc.slice/crio-aac7a9a41a119c49c3d26f162d40a63988bb389ede4c9f7a04bf40a6ff7c93ef WatchSource:0}: Error finding container aac7a9a41a119c49c3d26f162d40a63988bb389ede4c9f7a04bf40a6ff7c93ef: Status 404 returned error can't find the container with id aac7a9a41a119c49c3d26f162d40a63988bb389ede4c9f7a04bf40a6ff7c93ef Jul 09 04:08:05.420272 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.420234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:05.551345 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:05.551313 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:08:05.554636 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:08:05.554609 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b93b5c_308c_40f5_bd93_e786fda56977.slice/crio-346a99d370445743807dfd07baf6c0cf84f1a852dc52ebb3764f8ffd434457a5 WatchSource:0}: Error finding container 346a99d370445743807dfd07baf6c0cf84f1a852dc52ebb3764f8ffd434457a5: Status 404 returned error can't find the container with id 346a99d370445743807dfd07baf6c0cf84f1a852dc52ebb3764f8ffd434457a5 Jul 09 04:08:06.371707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:06.371624 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" event={"ID":"68b93b5c-308c-40f5-bd93-e786fda56977","Type":"ContainerStarted","Data":"346a99d370445743807dfd07baf6c0cf84f1a852dc52ebb3764f8ffd434457a5"} Jul 09 04:08:06.373247 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:06.373212 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c555c58c-424lj" event={"ID":"a4498532-947d-4d29-99b3-b2f0605f71fc","Type":"ContainerStarted","Data":"aac7a9a41a119c49c3d26f162d40a63988bb389ede4c9f7a04bf40a6ff7c93ef"} Jul 09 04:08:08.385930 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.385879 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" event={"ID":"68b93b5c-308c-40f5-bd93-e786fda56977","Type":"ContainerStarted","Data":"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d"} Jul 09 04:08:08.386437 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.386087 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:08.387519 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.387493 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c555c58c-424lj" event={"ID":"a4498532-947d-4d29-99b3-b2f0605f71fc","Type":"ContainerStarted","Data":"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73"} Jul 09 04:08:08.387632 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.387539 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:08.406648 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.406594 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" podStartSLOduration=1.4270065029999999 podStartE2EDuration="3.406572014s" totalImagesPullingTime="1.979565511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:08.405802451 +0000 UTC m=+488.009761194" watchObservedRunningTime="2026-07-09 04:08:08.406572014 +0000 UTC m=+488.010530755" Jul 09 04:08:08.425296 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:08.425237 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54c555c58c-424lj" podStartSLOduration=2.28308878 podStartE2EDuration="4.425217687s" totalImagesPullingTime="2.142128907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:08.422970218 +0000 UTC m=+488.026928954" watchObservedRunningTime="2026-07-09 04:08:08.425217687 +0000 UTC m=+488.029176429" Jul 09 04:08:19.399274 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.399236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:19.399883 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.399325 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:08:19.456851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.456818 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:19.457042 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.457015 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-54c555c58c-424lj" podUID="a4498532-947d-4d29-99b3-b2f0605f71fc" containerName="manager" containerID="cri-o://0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73" gracePeriod=30 Jul 09 04:08:19.705306 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.705284 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:19.774693 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.774666 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndsmk\" (UniqueName: \"kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk\") pod \"a4498532-947d-4d29-99b3-b2f0605f71fc\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " Jul 09 04:08:19.774850 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.774719 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert\") pod \"a4498532-947d-4d29-99b3-b2f0605f71fc\" (UID: \"a4498532-947d-4d29-99b3-b2f0605f71fc\") " Jul 09 04:08:19.776605 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.776579 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk" (OuterVolumeSpecName: "kube-api-access-ndsmk") pod "a4498532-947d-4d29-99b3-b2f0605f71fc" (UID: "a4498532-947d-4d29-99b3-b2f0605f71fc"). InnerVolumeSpecName "kube-api-access-ndsmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:08:19.776709 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.776695 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a4498532-947d-4d29-99b3-b2f0605f71fc" (UID: "a4498532-947d-4d29-99b3-b2f0605f71fc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:08:19.876055 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.876031 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndsmk\" (UniqueName: \"kubernetes.io/projected/a4498532-947d-4d29-99b3-b2f0605f71fc-kube-api-access-ndsmk\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:08:19.876055 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:19.876052 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4498532-947d-4d29-99b3-b2f0605f71fc-webhook-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:08:20.458826 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.458786 2570 generic.go:356] "Generic (PLEG): container finished" podID="a4498532-947d-4d29-99b3-b2f0605f71fc" containerID="0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73" exitCode=0 Jul 09 04:08:20.459252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.458847 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c555c58c-424lj" Jul 09 04:08:20.459252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.458848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c555c58c-424lj" event={"ID":"a4498532-947d-4d29-99b3-b2f0605f71fc","Type":"ContainerDied","Data":"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73"} Jul 09 04:08:20.459252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.458987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c555c58c-424lj" event={"ID":"a4498532-947d-4d29-99b3-b2f0605f71fc","Type":"ContainerDied","Data":"aac7a9a41a119c49c3d26f162d40a63988bb389ede4c9f7a04bf40a6ff7c93ef"} Jul 09 04:08:20.459252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.459011 2570 scope.go:117] "RemoveContainer" containerID="0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73" Jul 09 04:08:20.468734 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.468716 2570 scope.go:117] "RemoveContainer" containerID="0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73" Jul 09 04:08:20.468982 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:08:20.468966 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73\": container with ID starting with 0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73 not found: ID does not exist" containerID="0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73" Jul 09 04:08:20.469021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.468990 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73"} err="failed to get container status \"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73\": rpc error: code = NotFound desc = could not find container \"0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73\": container with ID starting with 0c91a463a3bdaad0df3a632b71aea00cd016a5ad569461214c73098518bb2d73 not found: ID does not exist" Jul 09 04:08:20.479916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.479891 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:20.485804 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.485781 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54c555c58c-424lj"] Jul 09 04:08:20.920991 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:20.920958 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4498532-947d-4d29-99b3-b2f0605f71fc" path="/var/lib/kubelet/pods/a4498532-947d-4d29-99b3-b2f0605f71fc/volumes" Jul 09 04:08:40.706317 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.706284 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b49766fb5-65srr"] Jul 09 04:08:40.706723 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.706702 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4498532-947d-4d29-99b3-b2f0605f71fc" containerName="manager" Jul 09 04:08:40.706723 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.706717 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4498532-947d-4d29-99b3-b2f0605f71fc" containerName="manager" Jul 09 04:08:40.706794 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.706785 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4498532-947d-4d29-99b3-b2f0605f71fc" containerName="manager" Jul 09 04:08:40.714162 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.714138 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.717394 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.717371 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 04:08:40.717503 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.717377 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cfhx4\"" Jul 09 04:08:40.725796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.725775 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b49766fb5-65srr"] Jul 09 04:08:40.847186 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.847159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8401fa-7b94-4772-b75b-6ba80d4690b3-openshift-service-ca\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.847395 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.847209 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcsp\" (UniqueName: \"kubernetes.io/projected/0b8401fa-7b94-4772-b75b-6ba80d4690b3-kube-api-access-kfcsp\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.847395 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.847289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8401fa-7b94-4772-b75b-6ba80d4690b3-maas-api-tls\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.948191 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.948166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8401fa-7b94-4772-b75b-6ba80d4690b3-openshift-service-ca\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.948295 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.948208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcsp\" (UniqueName: \"kubernetes.io/projected/0b8401fa-7b94-4772-b75b-6ba80d4690b3-kube-api-access-kfcsp\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.948295 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.948253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8401fa-7b94-4772-b75b-6ba80d4690b3-maas-api-tls\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.948847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.948823 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8401fa-7b94-4772-b75b-6ba80d4690b3-openshift-service-ca\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.950542 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.950522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8401fa-7b94-4772-b75b-6ba80d4690b3-maas-api-tls\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:40.955396 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:40.955376 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcsp\" (UniqueName: \"kubernetes.io/projected/0b8401fa-7b94-4772-b75b-6ba80d4690b3-kube-api-access-kfcsp\") pod \"maas-api-7b49766fb5-65srr\" (UID: \"0b8401fa-7b94-4772-b75b-6ba80d4690b3\") " pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:41.025847 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.025797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:41.165549 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.165517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b49766fb5-65srr"] Jul 09 04:08:41.166670 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:08:41.166639 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8401fa_7b94_4772_b75b_6ba80d4690b3.slice/crio-e10f43c130a2c30f6d454f073fceffb9e9d2e5d8fb57707c43809e207d4059f9 WatchSource:0}: Error finding container e10f43c130a2c30f6d454f073fceffb9e9d2e5d8fb57707c43809e207d4059f9: Status 404 returned error can't find the container with id e10f43c130a2c30f6d454f073fceffb9e9d2e5d8fb57707c43809e207d4059f9 Jul 09 04:08:41.309950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.309922 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ts96l"] Jul 09 04:08:41.312853 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.312837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.314794 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.314774 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 04:08:41.314906 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.314837 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pxrp4\"" Jul 09 04:08:41.321066 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.321044 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ts96l"] Jul 09 04:08:41.351419 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.351394 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f60-4870-4735-a23c-cee19409ccbf-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.351537 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.351458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4l89\" (UniqueName: \"kubernetes.io/projected/3a713f60-4870-4735-a23c-cee19409ccbf-kube-api-access-q4l89\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.452443 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.452388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f60-4870-4735-a23c-cee19409ccbf-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.452608 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.452496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4l89\" (UniqueName: \"kubernetes.io/projected/3a713f60-4870-4735-a23c-cee19409ccbf-kube-api-access-q4l89\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.453133 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.453112 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f60-4870-4735-a23c-cee19409ccbf-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.459714 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.459693 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4l89\" (UniqueName: \"kubernetes.io/projected/3a713f60-4870-4735-a23c-cee19409ccbf-kube-api-access-q4l89\") pod \"payload-pre-processing-7964457696-ts96l\" (UID: \"3a713f60-4870-4735-a23c-cee19409ccbf\") " pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.567220 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.567136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" event={"ID":"0b8401fa-7b94-4772-b75b-6ba80d4690b3","Type":"ContainerStarted","Data":"e10f43c130a2c30f6d454f073fceffb9e9d2e5d8fb57707c43809e207d4059f9"} Jul 09 04:08:41.625890 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.625862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:41.750560 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:41.750536 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ts96l"] Jul 09 04:08:41.752410 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:08:41.752383 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a713f60_4870_4735_a23c_cee19409ccbf.slice/crio-78845a48c5e5082e4729d6e69b97cde49a41eaf50b61e2165a7f1fc47a0d8408 WatchSource:0}: Error finding container 78845a48c5e5082e4729d6e69b97cde49a41eaf50b61e2165a7f1fc47a0d8408: Status 404 returned error can't find the container with id 78845a48c5e5082e4729d6e69b97cde49a41eaf50b61e2165a7f1fc47a0d8408 Jul 09 04:08:42.593372 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:42.590591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" event={"ID":"3a713f60-4870-4735-a23c-cee19409ccbf","Type":"ContainerStarted","Data":"78845a48c5e5082e4729d6e69b97cde49a41eaf50b61e2165a7f1fc47a0d8408"} Jul 09 04:08:43.600915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.599960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" event={"ID":"0b8401fa-7b94-4772-b75b-6ba80d4690b3","Type":"ContainerStarted","Data":"bd2da8efe98a91116b9715ca1732a73988695de2b5b3830748e17e5429a69ac1"} Jul 09 04:08:43.600915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.600862 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:43.603258 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.603217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" event={"ID":"3a713f60-4870-4735-a23c-cee19409ccbf","Type":"ContainerStarted","Data":"06f259ecd20988c306c3ad753a85a129c4df01e89bb61923b50ed8a52500a36e"} Jul 09 04:08:43.603901 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.603882 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:08:43.615677 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.615637 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" podStartSLOduration=1.374881603 podStartE2EDuration="3.615625141s" totalImagesPullingTime="2.240743538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:43.61447148 +0000 UTC m=+523.218430222" watchObservedRunningTime="2026-07-09 04:08:43.615625141 +0000 UTC m=+523.219583939" Jul 09 04:08:43.629214 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:43.629177 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" podStartSLOduration=0.972775298 podStartE2EDuration="2.629166498s" totalImagesPullingTime="1.6563912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:43.627038107 +0000 UTC m=+523.230996861" watchObservedRunningTime="2026-07-09 04:08:43.629166498 +0000 UTC m=+523.233125236" Jul 09 04:08:49.638185 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.638155 2570 generic.go:356] "Generic (PLEG): container finished" podID="a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c" containerID="481c4556d3e3332fcb9e1b355612d9ee29bf805fdd852c5b749185ac24a57e6f" exitCode=0 Jul 09 04:08:49.638614 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.638234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" event={"ID":"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c","Type":"ContainerDied","Data":"481c4556d3e3332fcb9e1b355612d9ee29bf805fdd852c5b749185ac24a57e6f"} Jul 09 04:08:49.638614 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.638588 2570 scope.go:117] "RemoveContainer" containerID="481c4556d3e3332fcb9e1b355612d9ee29bf805fdd852c5b749185ac24a57e6f" Jul 09 04:08:49.639544 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.639509 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:08:49.639609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.639590 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:08:49.639652 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:49.639627 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 04:08:50.619392 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:50.619337 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b49766fb5-65srr" Jul 09 04:08:50.647276 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:50.647244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" event={"ID":"a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c","Type":"ContainerStarted","Data":"c1c5cb387f03ef230f2573e35d64e86692976bcbd76e585ef17afd9e916a70df"} Jul 09 04:08:50.665542 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:50.665514 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:08:50.666472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:50.666442 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-tmvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 04:08:50.666612 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:50.666500 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" podUID="a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 04:08:51.666563 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:51.666528 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-tmvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 04:08:51.666966 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:51.666582 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" podUID="a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 04:08:52.666410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:52.666377 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-tmvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 04:08:52.666598 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:52.666428 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" podUID="a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 04:08:53.666608 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:53.666528 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-tmvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 04:08:53.666608 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:53.666592 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" podUID="a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 04:08:54.669934 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:54.669904 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:08:54.670371 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:54.670303 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:08:54.671030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:54.671014 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-tmvfs" Jul 09 04:08:55.615915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:08:55.615889 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-ts96l" Jul 09 04:09:04.972156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.972124 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9"] Jul 09 04:09:04.975509 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.975487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:04.978386 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.978339 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 04:09:04.978501 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.978346 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j7ms6\"" Jul 09 04:09:04.978501 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.978344 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 04:09:04.979054 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.979035 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 04:09:04.997557 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:04.997537 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9"] Jul 09 04:09:05.058088 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.058219 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b7e496-af03-446f-8da4-1c494e82030d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.058219 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.058337 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.058337 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.058470 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.058338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/a6b7e496-af03-446f-8da4-1c494e82030d-kube-api-access-7nn5v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.159797 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.159769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.159953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.159801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/a6b7e496-af03-446f-8da4-1c494e82030d-kube-api-access-7nn5v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.159953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.159837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.159953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.159866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b7e496-af03-446f-8da4-1c494e82030d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.160118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.159971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.160118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.160003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.160228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.160205 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.160282 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.160272 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.160379 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.160341 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.162076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.162056 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b7e496-af03-446f-8da4-1c494e82030d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.162248 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.162232 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b7e496-af03-446f-8da4-1c494e82030d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.167645 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.167620 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/a6b7e496-af03-446f-8da4-1c494e82030d-kube-api-access-7nn5v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9\" (UID: \"a6b7e496-af03-446f-8da4-1c494e82030d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.286779 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.286717 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:05.418836 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.418812 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9"] Jul 09 04:09:05.420398 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:09:05.420363 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b7e496_af03_446f_8da4_1c494e82030d.slice/crio-f727e0213c9ee4b01bd942df9ed95e851c69887b4b6f0d576acb1108ac82c4df WatchSource:0}: Error finding container f727e0213c9ee4b01bd942df9ed95e851c69887b4b6f0d576acb1108ac82c4df: Status 404 returned error can't find the container with id f727e0213c9ee4b01bd942df9ed95e851c69887b4b6f0d576acb1108ac82c4df Jul 09 04:09:05.735228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:05.735176 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" event={"ID":"a6b7e496-af03-446f-8da4-1c494e82030d","Type":"ContainerStarted","Data":"f727e0213c9ee4b01bd942df9ed95e851c69887b4b6f0d576acb1108ac82c4df"} Jul 09 04:09:06.742204 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:06.742126 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" event={"ID":"a6b7e496-af03-446f-8da4-1c494e82030d","Type":"ContainerStarted","Data":"bab509e6fa32ba5d8bb4cac68d2bc9b94fb730a0b1b6e4b9e2763d993149bdbd"} Jul 09 04:09:06.762553 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:06.762510 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" podStartSLOduration=1.7370112359999998 podStartE2EDuration="2.762496156s" totalImagesPullingTime="1.02548492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:09:06.761342987 +0000 UTC m=+546.365301726" watchObservedRunningTime="2026-07-09 04:09:06.762496156 +0000 UTC m=+546.366454897" Jul 09 04:09:15.287297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:15.287254 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:15.287768 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:15.287329 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:15.299801 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:15.299764 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:15.803628 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:15.803596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9" Jul 09 04:09:25.839217 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.839173 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dph76"] Jul 09 04:09:25.843439 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.843408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.845765 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.845745 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 04:09:25.851991 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.851957 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dph76"] Jul 09 04:09:25.965342 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.965574 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.965574 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb33c918-775b-47a5-ad21-5ba4526057e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.965574 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhqc\" (UniqueName: \"kubernetes.io/projected/bb33c918-775b-47a5-ad21-5ba4526057e4-kube-api-access-thhqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.965574 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:25.965727 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:25.965640 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066226 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066179 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb33c918-775b-47a5-ad21-5ba4526057e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhqc\" (UniqueName: \"kubernetes.io/projected/bb33c918-775b-47a5-ad21-5ba4526057e4-kube-api-access-thhqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066453 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066753 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066816 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066774 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.066857 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.066846 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.068655 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.068632 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb33c918-775b-47a5-ad21-5ba4526057e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.068831 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.068814 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb33c918-775b-47a5-ad21-5ba4526057e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.074813 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.074789 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhqc\" (UniqueName: \"kubernetes.io/projected/bb33c918-775b-47a5-ad21-5ba4526057e4-kube-api-access-thhqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dph76\" (UID: \"bb33c918-775b-47a5-ad21-5ba4526057e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.155250 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.155144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:26.312603 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.312573 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dph76"] Jul 09 04:09:26.314662 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:09:26.314630 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb33c918_775b_47a5_ad21_5ba4526057e4.slice/crio-4898ed7dab54539b1bafe3cd8e64578749bc71e6efef17d6f11c54f134e04ae4 WatchSource:0}: Error finding container 4898ed7dab54539b1bafe3cd8e64578749bc71e6efef17d6f11c54f134e04ae4: Status 404 returned error can't find the container with id 4898ed7dab54539b1bafe3cd8e64578749bc71e6efef17d6f11c54f134e04ae4 Jul 09 04:09:26.852642 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.852605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" event={"ID":"bb33c918-775b-47a5-ad21-5ba4526057e4","Type":"ContainerStarted","Data":"744be3f6259f56d10bbe7dad69f11dbfc59893080692e493ab5d18e19986c9b8"} Jul 09 04:09:26.853027 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.852659 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" event={"ID":"bb33c918-775b-47a5-ad21-5ba4526057e4","Type":"ContainerStarted","Data":"4898ed7dab54539b1bafe3cd8e64578749bc71e6efef17d6f11c54f134e04ae4"} Jul 09 04:09:26.871177 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:26.871122 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" podStartSLOduration=1.6698791320000002 podStartE2EDuration="1.871103807s" totalImagesPullingTime="201.224675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:09:26.870148691 +0000 UTC m=+566.474107451" watchObservedRunningTime="2026-07-09 04:09:26.871103807 +0000 UTC m=+566.475062547" Jul 09 04:09:36.156310 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:36.156264 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:36.156833 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:36.156334 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:36.169297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:36.169272 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:36.921796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:36.921765 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dph76" Jul 09 04:09:37.048298 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.048262 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj"] Jul 09 04:09:37.052146 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.052124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.054400 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.054340 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 04:09:37.060712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.060676 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj"] Jul 09 04:09:37.073633 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.073799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073640 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.073799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.073967 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.073967 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdsv\" (UniqueName: \"kubernetes.io/projected/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-kube-api-access-6xdsv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.073967 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.073902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.174697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdsv\" (UniqueName: \"kubernetes.io/projected/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-kube-api-access-6xdsv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.174697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.174697 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.174794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.175186 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.175210 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.175265 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.175246 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.177166 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.177138 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.177867 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.177842 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.185110 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.185078 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdsv\" (UniqueName: \"kubernetes.io/projected/f8f1bfe5-d85e-4c5f-9ea9-66196812f3da-kube-api-access-6xdsv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj\" (UID: \"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.364934 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.364895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:37.516858 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.516831 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj"] Jul 09 04:09:37.519526 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:09:37.519492 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f1bfe5_d85e_4c5f_9ea9_66196812f3da.slice/crio-d6839f3db7ac07acc1ccd6c2d694c2caa700b301095392ff77c9f87459cb723a WatchSource:0}: Error finding container d6839f3db7ac07acc1ccd6c2d694c2caa700b301095392ff77c9f87459cb723a: Status 404 returned error can't find the container with id d6839f3db7ac07acc1ccd6c2d694c2caa700b301095392ff77c9f87459cb723a Jul 09 04:09:37.916567 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.916523 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" event={"ID":"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da","Type":"ContainerStarted","Data":"026e6ef37b3084fb9b5393f8f67458c46c53fe62c4856874487a4cf25ae72b8d"} Jul 09 04:09:37.916777 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.916579 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" event={"ID":"f8f1bfe5-d85e-4c5f-9ea9-66196812f3da","Type":"ContainerStarted","Data":"d6839f3db7ac07acc1ccd6c2d694c2caa700b301095392ff77c9f87459cb723a"} Jul 09 04:09:37.934076 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:37.934026 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" podStartSLOduration=0.748690833 podStartE2EDuration="934.00717ms" totalImagesPullingTime="185.316337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:09:37.931736897 +0000 UTC m=+577.535695667" watchObservedRunningTime="2026-07-09 04:09:37.93400717 +0000 UTC m=+577.537965911" Jul 09 04:09:47.365476 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:47.365436 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:47.365915 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:47.365538 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:47.378651 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:47.378617 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:09:47.983057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:09:47.983019 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj" Jul 09 04:10:03.036620 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:10:03.036589 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:10:03.039755 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:10:03.039729 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:12:33.314248 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.314209 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:12:33.314849 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.314462 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" podUID="68b93b5c-308c-40f5-bd93-e786fda56977" containerName="manager" containerID="cri-o://26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d" gracePeriod=30 Jul 09 04:12:33.576773 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.576704 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:12:33.647855 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.647820 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert\") pod \"68b93b5c-308c-40f5-bd93-e786fda56977\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " Jul 09 04:12:33.648021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.647914 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj6h9\" (UniqueName: \"kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9\") pod \"68b93b5c-308c-40f5-bd93-e786fda56977\" (UID: \"68b93b5c-308c-40f5-bd93-e786fda56977\") " Jul 09 04:12:33.649982 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.649948 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68b93b5c-308c-40f5-bd93-e786fda56977" (UID: "68b93b5c-308c-40f5-bd93-e786fda56977"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:12:33.650120 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.649998 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9" (OuterVolumeSpecName: "kube-api-access-tj6h9") pod "68b93b5c-308c-40f5-bd93-e786fda56977" (UID: "68b93b5c-308c-40f5-bd93-e786fda56977"). InnerVolumeSpecName "kube-api-access-tj6h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:12:33.749027 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.748986 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68b93b5c-308c-40f5-bd93-e786fda56977-webhook-cert\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:12:33.749027 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.749019 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj6h9\" (UniqueName: \"kubernetes.io/projected/68b93b5c-308c-40f5-bd93-e786fda56977-kube-api-access-tj6h9\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:12:33.878783 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.878686 2570 generic.go:356] "Generic (PLEG): container finished" podID="68b93b5c-308c-40f5-bd93-e786fda56977" containerID="26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d" exitCode=0 Jul 09 04:12:33.878783 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.878735 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" Jul 09 04:12:33.878783 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.878770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" event={"ID":"68b93b5c-308c-40f5-bd93-e786fda56977","Type":"ContainerDied","Data":"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d"} Jul 09 04:12:33.879030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.878822 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-kvgrj" event={"ID":"68b93b5c-308c-40f5-bd93-e786fda56977","Type":"ContainerDied","Data":"346a99d370445743807dfd07baf6c0cf84f1a852dc52ebb3764f8ffd434457a5"} Jul 09 04:12:33.879030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.878844 2570 scope.go:117] "RemoveContainer" containerID="26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d" Jul 09 04:12:33.888731 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.888709 2570 scope.go:117] "RemoveContainer" containerID="26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d" Jul 09 04:12:33.889071 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:12:33.889050 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d\": container with ID starting with 26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d not found: ID does not exist" containerID="26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d" Jul 09 04:12:33.889138 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.889083 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d"} err="failed to get container status \"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d\": rpc error: code = NotFound desc = could not find container \"26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d\": container with ID starting with 26922bd07deb83d77ee650a8d7b70de38258fb4e7694739e868425c62859b50d not found: ID does not exist" Jul 09 04:12:33.900690 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.900655 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:12:33.904327 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:33.904298 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-kvgrj"] Jul 09 04:12:34.921614 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:12:34.921570 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b93b5c-308c-40f5-bd93-e786fda56977" path="/var/lib/kubelet/pods/68b93b5c-308c-40f5-bd93-e786fda56977/volumes" Jul 09 04:13:01.386429 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:13:01.386395 2570 scope.go:117] "RemoveContainer" containerID="59c5a2201a51722c778cb5ea9e61fcd3ea5e0cfcc980cb2103b504d141952666" Jul 09 04:15:03.070266 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:15:03.070227 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:15:03.075313 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:15:03.075289 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:17:25.835394 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:25.835342 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:17:25.835881 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:25.835569 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" podUID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" containerName="manager" containerID="cri-o://181fda6be879231a2690bc152f63b713a3dde2a3b09eb9572a6339a3ec110210" gracePeriod=10 Jul 09 04:17:28.502405 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.502371 2570 generic.go:356] "Generic (PLEG): container finished" podID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" containerID="181fda6be879231a2690bc152f63b713a3dde2a3b09eb9572a6339a3ec110210" exitCode=0 Jul 09 04:17:28.502405 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.502387 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" event={"ID":"ad46da13-5e1b-408d-8e2d-91a89eaa4936","Type":"ContainerDied","Data":"181fda6be879231a2690bc152f63b713a3dde2a3b09eb9572a6339a3ec110210"} Jul 09 04:17:28.608994 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.608966 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:17:28.736937 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.736847 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume\") pod \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " Jul 09 04:17:28.737108 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.736939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klbt4\" (UniqueName: \"kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4\") pod \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\" (UID: \"ad46da13-5e1b-408d-8e2d-91a89eaa4936\") " Jul 09 04:17:28.737294 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.737268 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad46da13-5e1b-408d-8e2d-91a89eaa4936" (UID: "ad46da13-5e1b-408d-8e2d-91a89eaa4936"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:17:28.738945 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.738924 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4" (OuterVolumeSpecName: "kube-api-access-klbt4") pod "ad46da13-5e1b-408d-8e2d-91a89eaa4936" (UID: "ad46da13-5e1b-408d-8e2d-91a89eaa4936"). InnerVolumeSpecName "kube-api-access-klbt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:17:28.838380 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.838330 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbt4\" (UniqueName: \"kubernetes.io/projected/ad46da13-5e1b-408d-8e2d-91a89eaa4936-kube-api-access-klbt4\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:17:28.838380 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:28.838374 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad46da13-5e1b-408d-8e2d-91a89eaa4936-extensions-socket-volume\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:17:29.509007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:29.508976 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" Jul 09 04:17:29.509007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:29.508987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh" event={"ID":"ad46da13-5e1b-408d-8e2d-91a89eaa4936","Type":"ContainerDied","Data":"63d33cc2b122499f9dc713c6056650a0a6abb8235b58fb0a7b4487f76a42ef5c"} Jul 09 04:17:29.509531 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:29.509054 2570 scope.go:117] "RemoveContainer" containerID="181fda6be879231a2690bc152f63b713a3dde2a3b09eb9572a6339a3ec110210" Jul 09 04:17:29.525103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:29.525076 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:17:29.529066 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:29.529045 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9rsh"] Jul 09 04:17:30.922166 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:17:30.922132 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" path="/var/lib/kubelet/pods/ad46da13-5e1b-408d-8e2d-91a89eaa4936/volumes" Jul 09 04:18:26.962445 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962406 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz"] Jul 09 04:18:26.962866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962833 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" containerName="manager" Jul 09 04:18:26.962866 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962844 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" containerName="manager" Jul 09 04:18:26.962950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962873 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b93b5c-308c-40f5-bd93-e786fda56977" containerName="manager" Jul 09 04:18:26.962950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962879 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b93b5c-308c-40f5-bd93-e786fda56977" containerName="manager" Jul 09 04:18:26.962950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962946 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b93b5c-308c-40f5-bd93-e786fda56977" containerName="manager" Jul 09 04:18:26.963042 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.962956 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad46da13-5e1b-408d-8e2d-91a89eaa4936" containerName="manager" Jul 09 04:18:26.967128 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.967107 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:26.969510 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.969492 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wkj2r\"" Jul 09 04:18:26.979863 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:26.979834 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz"] Jul 09 04:18:27.156501 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.156463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aa85859-28fd-4299-a303-f2a0228c1b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.156682 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.156522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrm2t\" (UniqueName: \"kubernetes.io/projected/2aa85859-28fd-4299-a303-f2a0228c1b59-kube-api-access-hrm2t\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.257336 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.257236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aa85859-28fd-4299-a303-f2a0228c1b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.257336 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.257301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrm2t\" (UniqueName: \"kubernetes.io/projected/2aa85859-28fd-4299-a303-f2a0228c1b59-kube-api-access-hrm2t\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.257745 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.257725 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aa85859-28fd-4299-a303-f2a0228c1b59-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.265855 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.265832 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrm2t\" (UniqueName: \"kubernetes.io/projected/2aa85859-28fd-4299-a303-f2a0228c1b59-kube-api-access-hrm2t\") pod \"kuadrant-operator-controller-manager-86878c769f-hd2mz\" (UID: \"2aa85859-28fd-4299-a303-f2a0228c1b59\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.278818 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.278787 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.446131 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.446107 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz"] Jul 09 04:18:27.447962 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:18:27.447932 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa85859_28fd_4299_a303_f2a0228c1b59.slice/crio-15e9d21c4f28be675e209a5d73592fb1445d994eecb1909ac98660bef5804857 WatchSource:0}: Error finding container 15e9d21c4f28be675e209a5d73592fb1445d994eecb1909ac98660bef5804857: Status 404 returned error can't find the container with id 15e9d21c4f28be675e209a5d73592fb1445d994eecb1909ac98660bef5804857 Jul 09 04:18:27.816044 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.816005 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" event={"ID":"2aa85859-28fd-4299-a303-f2a0228c1b59","Type":"ContainerStarted","Data":"2e63c92b895ae9772ce63fad53d66c2ef740c5df3a8d3fffd920588f600af98f"} Jul 09 04:18:27.816218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.816066 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" event={"ID":"2aa85859-28fd-4299-a303-f2a0228c1b59","Type":"ContainerStarted","Data":"15e9d21c4f28be675e209a5d73592fb1445d994eecb1909ac98660bef5804857"} Jul 09 04:18:27.816218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.816098 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:18:27.836429 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:27.836374 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" podStartSLOduration=1.836340421 podStartE2EDuration="1.836340421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:18:27.834040311 +0000 UTC m=+1107.437999074" watchObservedRunningTime="2026-07-09 04:18:27.836340421 +0000 UTC m=+1107.440299161" Jul 09 04:18:38.824344 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:18:38.824312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hd2mz" Jul 09 04:20:03.104518 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:20:03.104482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:20:03.111561 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:20:03.111537 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:24:50.414369 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.414321 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:50.418609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.418584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.420667 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.420644 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b4276fe1-serving-cert\"" Jul 09 04:24:50.427323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.427301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:50.598873 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.598832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.599054 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.598901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.599054 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.598957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.700244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.700139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.700436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.700246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.700436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.700405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.701084 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.701062 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.702861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.702830 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.707617 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.707589 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr\") pod \"maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.730192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.730167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:50.874836 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.874805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:50.877442 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:24:50.877397 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabc6392_091d_4761_a38c_7e04de3b3e1c.slice/crio-6d6e230b1dcbcf0710b71b5f11ba68c4e9a488572cfd166803d18c02bbaa8df6 WatchSource:0}: Error finding container 6d6e230b1dcbcf0710b71b5f11ba68c4e9a488572cfd166803d18c02bbaa8df6: Status 404 returned error can't find the container with id 6d6e230b1dcbcf0710b71b5f11ba68c4e9a488572cfd166803d18c02bbaa8df6 Jul 09 04:24:50.879326 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:50.879304 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:24:51.856373 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:51.856316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" event={"ID":"eabc6392-091d-4761-a38c-7e04de3b3e1c","Type":"ContainerStarted","Data":"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122"} Jul 09 04:24:51.856761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:51.856392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" event={"ID":"eabc6392-091d-4761-a38c-7e04de3b3e1c","Type":"ContainerStarted","Data":"6d6e230b1dcbcf0710b71b5f11ba68c4e9a488572cfd166803d18c02bbaa8df6"} Jul 09 04:24:51.856761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:51.856424 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:51.872927 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:51.872861 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" podStartSLOduration=1.429491375 podStartE2EDuration="1.872845339s" totalImagesPullingTime="443.353964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:24:51.870398078 +0000 UTC m=+1491.474356857" watchObservedRunningTime="2026-07-09 04:24:51.872845339 +0000 UTC m=+1491.476804093" Jul 09 04:24:55.592416 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.592380 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:55.592962 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.592625 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" podUID="eabc6392-091d-4761-a38c-7e04de3b3e1c" containerName="maas-api" containerID="cri-o://0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122" gracePeriod=30 Jul 09 04:24:55.860399 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.860373 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:55.882926 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.882896 2570 generic.go:356] "Generic (PLEG): container finished" podID="eabc6392-091d-4761-a38c-7e04de3b3e1c" containerID="0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122" exitCode=0 Jul 09 04:24:55.883097 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.883073 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" event={"ID":"eabc6392-091d-4761-a38c-7e04de3b3e1c","Type":"ContainerDied","Data":"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122"} Jul 09 04:24:55.883185 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.883129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" event={"ID":"eabc6392-091d-4761-a38c-7e04de3b3e1c","Type":"ContainerDied","Data":"6d6e230b1dcbcf0710b71b5f11ba68c4e9a488572cfd166803d18c02bbaa8df6"} Jul 09 04:24:55.883185 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.883166 2570 scope.go:117] "RemoveContainer" containerID="0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122" Jul 09 04:24:55.883384 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.883338 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn" Jul 09 04:24:55.893262 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.893240 2570 scope.go:117] "RemoveContainer" containerID="0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122" Jul 09 04:24:55.893557 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:24:55.893534 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122\": container with ID starting with 0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122 not found: ID does not exist" containerID="0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122" Jul 09 04:24:55.893639 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.893565 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122"} err="failed to get container status \"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122\": rpc error: code = NotFound desc = could not find container \"0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122\": container with ID starting with 0bd90335b2a0c3c17feaa81d75e07264e0ed4f792e21dec4238a907c913a7122 not found: ID does not exist" Jul 09 04:24:55.948895 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.948865 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca\") pod \"eabc6392-091d-4761-a38c-7e04de3b3e1c\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " Jul 09 04:24:55.949043 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.948914 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls\") pod \"eabc6392-091d-4761-a38c-7e04de3b3e1c\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " Jul 09 04:24:55.949043 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.949004 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr\") pod \"eabc6392-091d-4761-a38c-7e04de3b3e1c\" (UID: \"eabc6392-091d-4761-a38c-7e04de3b3e1c\") " Jul 09 04:24:55.949208 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.949185 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eabc6392-091d-4761-a38c-7e04de3b3e1c" (UID: "eabc6392-091d-4761-a38c-7e04de3b3e1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:24:55.951175 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.951152 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eabc6392-091d-4761-a38c-7e04de3b3e1c" (UID: "eabc6392-091d-4761-a38c-7e04de3b3e1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:24:55.951175 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:55.951166 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr" (OuterVolumeSpecName: "kube-api-access-vprrr") pod "eabc6392-091d-4761-a38c-7e04de3b3e1c" (UID: "eabc6392-091d-4761-a38c-7e04de3b3e1c"). InnerVolumeSpecName "kube-api-access-vprrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:24:56.049856 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.049823 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc6392-091d-4761-a38c-7e04de3b3e1c-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:24:56.049856 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.049853 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/eabc6392-091d-4761-a38c-7e04de3b3e1c-kube-api-access-vprrr\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:24:56.049856 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.049864 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc6392-091d-4761-a38c-7e04de3b3e1c-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:24:56.205383 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.205332 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:56.207373 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.207336 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4276fe1-5487878df7-qtsmn"] Jul 09 04:24:56.922039 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:24:56.922000 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabc6392-091d-4761-a38c-7e04de3b3e1c" path="/var/lib/kubelet/pods/eabc6392-091d-4761-a38c-7e04de3b3e1c/volumes" Jul 09 04:25:03.140632 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.140507 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:25:03.155322 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.149439 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:25:03.217892 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.217858 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:03.218406 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.218386 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabc6392-091d-4761-a38c-7e04de3b3e1c" containerName="maas-api" Jul 09 04:25:03.218406 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.218406 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabc6392-091d-4761-a38c-7e04de3b3e1c" containerName="maas-api" Jul 09 04:25:03.218641 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.218626 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabc6392-091d-4761-a38c-7e04de3b3e1c" containerName="maas-api" Jul 09 04:25:03.221963 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.221944 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.223844 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.223822 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-278ee172-serving-cert\"" Jul 09 04:25:03.232151 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.232127 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:03.320505 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.320468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnw2d\" (UniqueName: \"kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.320684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.320635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.320732 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.320685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.421530 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.421452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.421530 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.421494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.421708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.421534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw2d\" (UniqueName: \"kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.421708 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:03.421612 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-serving-cert: secret "maas-api-e2e-ait-278ee172-serving-cert" not found Jul 09 04:25:03.421708 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:03.421688 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls podName:93f90b45-1ff0-469d-823a-2e7f9f94f102 nodeName:}" failed. No retries permitted until 2026-07-09 04:25:03.921666676 +0000 UTC m=+1503.525625415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls") pod "maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" (UID: "93f90b45-1ff0-469d-823a-2e7f9f94f102") : secret "maas-api-e2e-ait-278ee172-serving-cert" not found Jul 09 04:25:03.422191 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.422172 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.430480 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.430454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw2d\" (UniqueName: \"kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.926699 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.926655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:03.929238 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:03.929213 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") pod \"maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:04.133494 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.133453 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:04.275938 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.275901 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:04.278043 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:25:04.278009 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f90b45_1ff0_469d_823a_2e7f9f94f102.slice/crio-e6057d989023bac74471554ec55c96a7bbe5a4db97eef504b146c8f5e1a72063 WatchSource:0}: Error finding container e6057d989023bac74471554ec55c96a7bbe5a4db97eef504b146c8f5e1a72063: Status 404 returned error can't find the container with id e6057d989023bac74471554ec55c96a7bbe5a4db97eef504b146c8f5e1a72063 Jul 09 04:25:04.946249 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.946204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" event={"ID":"93f90b45-1ff0-469d-823a-2e7f9f94f102","Type":"ContainerStarted","Data":"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154"} Jul 09 04:25:04.946488 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.946297 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" event={"ID":"93f90b45-1ff0-469d-823a-2e7f9f94f102","Type":"ContainerStarted","Data":"e6057d989023bac74471554ec55c96a7bbe5a4db97eef504b146c8f5e1a72063"} Jul 09 04:25:04.946488 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.946329 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:04.962430 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:04.962328 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" podStartSLOduration=1.541732313 podStartE2EDuration="1.96231326s" totalImagesPullingTime="420.580947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:25:04.960803926 +0000 UTC m=+1504.564762662" watchObservedRunningTime="2026-07-09 04:25:04.96231326 +0000 UTC m=+1504.566272002" Jul 09 04:25:08.524504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.524466 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:08.524984 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.524692 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" podUID="93f90b45-1ff0-469d-823a-2e7f9f94f102" containerName="maas-api" containerID="cri-o://d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154" gracePeriod=30 Jul 09 04:25:08.785507 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.785445 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:08.875449 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.875410 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca\") pod \"93f90b45-1ff0-469d-823a-2e7f9f94f102\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " Jul 09 04:25:08.875629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.875499 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") pod \"93f90b45-1ff0-469d-823a-2e7f9f94f102\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " Jul 09 04:25:08.875629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.875563 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnw2d\" (UniqueName: \"kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d\") pod \"93f90b45-1ff0-469d-823a-2e7f9f94f102\" (UID: \"93f90b45-1ff0-469d-823a-2e7f9f94f102\") " Jul 09 04:25:08.875872 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.875820 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93f90b45-1ff0-469d-823a-2e7f9f94f102" (UID: "93f90b45-1ff0-469d-823a-2e7f9f94f102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:25:08.877686 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.877651 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d" (OuterVolumeSpecName: "kube-api-access-dnw2d") pod "93f90b45-1ff0-469d-823a-2e7f9f94f102" (UID: "93f90b45-1ff0-469d-823a-2e7f9f94f102"). InnerVolumeSpecName "kube-api-access-dnw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:25:08.877686 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.877678 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93f90b45-1ff0-469d-823a-2e7f9f94f102" (UID: "93f90b45-1ff0-469d-823a-2e7f9f94f102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:25:08.967562 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.967529 2570 generic.go:356] "Generic (PLEG): container finished" podID="93f90b45-1ff0-469d-823a-2e7f9f94f102" containerID="d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154" exitCode=0 Jul 09 04:25:08.967708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.967573 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" Jul 09 04:25:08.967708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.967615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" event={"ID":"93f90b45-1ff0-469d-823a-2e7f9f94f102","Type":"ContainerDied","Data":"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154"} Jul 09 04:25:08.967708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.967668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2" event={"ID":"93f90b45-1ff0-469d-823a-2e7f9f94f102","Type":"ContainerDied","Data":"e6057d989023bac74471554ec55c96a7bbe5a4db97eef504b146c8f5e1a72063"} Jul 09 04:25:08.967708 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.967695 2570 scope.go:117] "RemoveContainer" containerID="d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154" Jul 09 04:25:08.976706 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.976687 2570 scope.go:117] "RemoveContainer" containerID="d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154" Jul 09 04:25:08.976973 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:08.976953 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154\": container with ID starting with d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154 not found: ID does not exist" containerID="d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154" Jul 09 04:25:08.977053 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.976981 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154"} err="failed to get container status \"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154\": rpc error: code = NotFound desc = could not find container \"d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154\": container with ID starting with d4d0a578f3dfc9288d89f9f0ec58addae20d07aaa60334587efe416fff85e154 not found: ID does not exist" Jul 09 04:25:08.977240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.977213 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f90b45-1ff0-469d-823a-2e7f9f94f102-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:08.977240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.977236 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f90b45-1ff0-469d-823a-2e7f9f94f102-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:08.977365 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.977255 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnw2d\" (UniqueName: \"kubernetes.io/projected/93f90b45-1ff0-469d-823a-2e7f9f94f102-kube-api-access-dnw2d\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:08.983731 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.983704 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:08.985810 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:08.985786 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-278ee172-c5dfb7bdf-tfkt2"] Jul 09 04:25:10.920541 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:10.920499 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f90b45-1ff0-469d-823a-2e7f9f94f102" path="/var/lib/kubelet/pods/93f90b45-1ff0-469d-823a-2e7f9f94f102/volumes" Jul 09 04:25:21.678479 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.678439 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:21.679168 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.679145 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f90b45-1ff0-469d-823a-2e7f9f94f102" containerName="maas-api" Jul 09 04:25:21.679168 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.679166 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f90b45-1ff0-469d-823a-2e7f9f94f102" containerName="maas-api" Jul 09 04:25:21.679309 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.679296 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f90b45-1ff0-469d-823a-2e7f9f94f102" containerName="maas-api" Jul 09 04:25:21.684209 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.684188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.686060 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.686034 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-aeb5be87-serving-cert\"" Jul 09 04:25:21.690030 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.689997 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:21.806884 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.806847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.807047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.806940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.807047 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.806984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.907643 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.907607 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.907807 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.907693 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.907807 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.907739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.907807 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:21.907763 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-serving-cert: secret "maas-api-e2e-derive-aeb5be87-serving-cert" not found Jul 09 04:25:21.907914 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:21.907828 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls podName:884ce39c-5c54-45c0-86e8-7ce9115aa2b4 nodeName:}" failed. No retries permitted until 2026-07-09 04:25:22.407812364 +0000 UTC m=+1522.011771083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls") pod "maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" (UID: "884ce39c-5c54-45c0-86e8-7ce9115aa2b4") : secret "maas-api-e2e-derive-aeb5be87-serving-cert" not found Jul 09 04:25:21.908325 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.908307 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:21.917320 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:21.917293 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:22.414003 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:22.413966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:22.416547 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:22.416517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") pod \"maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:22.597508 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:22.597472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:22.736803 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:22.736776 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:22.738633 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:25:22.738608 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884ce39c_5c54_45c0_86e8_7ce9115aa2b4.slice/crio-8aed14794c9a4e5db2eb1b21e19e63f397e91347b8ca7b3a93fd9f9c00e6c4aa WatchSource:0}: Error finding container 8aed14794c9a4e5db2eb1b21e19e63f397e91347b8ca7b3a93fd9f9c00e6c4aa: Status 404 returned error can't find the container with id 8aed14794c9a4e5db2eb1b21e19e63f397e91347b8ca7b3a93fd9f9c00e6c4aa Jul 09 04:25:23.060581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:23.060542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" event={"ID":"884ce39c-5c54-45c0-86e8-7ce9115aa2b4","Type":"ContainerStarted","Data":"8aed14794c9a4e5db2eb1b21e19e63f397e91347b8ca7b3a93fd9f9c00e6c4aa"} Jul 09 04:25:24.067911 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:24.067872 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" event={"ID":"884ce39c-5c54-45c0-86e8-7ce9115aa2b4","Type":"ContainerStarted","Data":"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17"} Jul 09 04:25:24.068405 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:24.067983 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:24.085230 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:24.085178 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" podStartSLOduration=2.736006532 podStartE2EDuration="3.085162814s" totalImagesPullingTime="349.156282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:25:24.083903431 +0000 UTC m=+1523.687862175" watchObservedRunningTime="2026-07-09 04:25:24.085162814 +0000 UTC m=+1523.689121555" Jul 09 04:25:26.376945 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.376909 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:26.377657 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.377624 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" podUID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" containerName="maas-api" containerID="cri-o://57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17" gracePeriod=30 Jul 09 04:25:26.630161 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.630094 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:26.756624 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.756589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca\") pod \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " Jul 09 04:25:26.756771 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.756638 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") pod \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " Jul 09 04:25:26.756771 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.756694 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm\") pod \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\" (UID: \"884ce39c-5c54-45c0-86e8-7ce9115aa2b4\") " Jul 09 04:25:26.756970 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.756946 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "884ce39c-5c54-45c0-86e8-7ce9115aa2b4" (UID: "884ce39c-5c54-45c0-86e8-7ce9115aa2b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:25:26.757079 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.757057 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:26.758752 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.758730 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm" (OuterVolumeSpecName: "kube-api-access-5hvcm") pod "884ce39c-5c54-45c0-86e8-7ce9115aa2b4" (UID: "884ce39c-5c54-45c0-86e8-7ce9115aa2b4"). InnerVolumeSpecName "kube-api-access-5hvcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:25:26.758836 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.758781 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "884ce39c-5c54-45c0-86e8-7ce9115aa2b4" (UID: "884ce39c-5c54-45c0-86e8-7ce9115aa2b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:25:26.858303 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.858270 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:26.858303 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:26.858298 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/884ce39c-5c54-45c0-86e8-7ce9115aa2b4-kube-api-access-5hvcm\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:25:27.086192 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.086159 2570 generic.go:356] "Generic (PLEG): container finished" podID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" containerID="57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17" exitCode=0 Jul 09 04:25:27.086344 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.086225 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" Jul 09 04:25:27.086344 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.086239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" event={"ID":"884ce39c-5c54-45c0-86e8-7ce9115aa2b4","Type":"ContainerDied","Data":"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17"} Jul 09 04:25:27.086344 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.086317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj" event={"ID":"884ce39c-5c54-45c0-86e8-7ce9115aa2b4","Type":"ContainerDied","Data":"8aed14794c9a4e5db2eb1b21e19e63f397e91347b8ca7b3a93fd9f9c00e6c4aa"} Jul 09 04:25:27.086344 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.086340 2570 scope.go:117] "RemoveContainer" containerID="57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17" Jul 09 04:25:27.095513 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.095490 2570 scope.go:117] "RemoveContainer" containerID="57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17" Jul 09 04:25:27.095775 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:25:27.095754 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17\": container with ID starting with 57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17 not found: ID does not exist" containerID="57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17" Jul 09 04:25:27.095842 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.095786 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17"} err="failed to get container status \"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17\": rpc error: code = NotFound desc = could not find container \"57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17\": container with ID starting with 57b8aa8474aa54374b9046670b007f6a1ee9fa7f7a24610c658751d5dda77e17 not found: ID does not exist" Jul 09 04:25:27.102397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.102372 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:27.106281 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:27.106261 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aeb5be87-78bc646558-lfvfj"] Jul 09 04:25:28.921404 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:25:28.921374 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" path="/var/lib/kubelet/pods/884ce39c-5c54-45c0-86e8-7ce9115aa2b4/volumes" Jul 09 04:27:10.671832 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.671753 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:10.672287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.672211 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" containerName="maas-api" Jul 09 04:27:10.672287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.672222 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" containerName="maas-api" Jul 09 04:27:10.672287 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.672286 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="884ce39c-5c54-45c0-86e8-7ce9115aa2b4" containerName="maas-api" Jul 09 04:27:10.675625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.675604 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.677759 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.677735 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3c4e5fa4-serving-cert\"" Jul 09 04:27:10.684711 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.684683 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:10.792796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.792758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5fg\" (UniqueName: \"kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.792992 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.792810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.792992 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.792942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.894603 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.894561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.894775 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.894654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5fg\" (UniqueName: \"kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.894775 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.894703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.894884 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:10.894868 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-serving-cert: secret "maas-api-e2e-mt-3c4e5fa4-serving-cert" not found Jul 09 04:27:10.894968 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:10.894957 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls podName:9a436569-84af-4899-aa75-37b6a6134b32 nodeName:}" failed. No retries permitted until 2026-07-09 04:27:11.39493296 +0000 UTC m=+1630.998891680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls") pod "maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" (UID: "9a436569-84af-4899-aa75-37b6a6134b32") : secret "maas-api-e2e-mt-3c4e5fa4-serving-cert" not found Jul 09 04:27:10.895235 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.895208 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:10.902635 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:10.902607 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5fg\" (UniqueName: \"kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:11.400223 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:11.400193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:11.400408 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:11.400336 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-serving-cert: secret "maas-api-e2e-mt-3c4e5fa4-serving-cert" not found Jul 09 04:27:11.400456 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:11.400423 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls podName:9a436569-84af-4899-aa75-37b6a6134b32 nodeName:}" failed. No retries permitted until 2026-07-09 04:27:12.400408773 +0000 UTC m=+1632.004367495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls") pod "maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" (UID: "9a436569-84af-4899-aa75-37b6a6134b32") : secret "maas-api-e2e-mt-3c4e5fa4-serving-cert" not found Jul 09 04:27:12.411776 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:12.411726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:12.414091 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:12.414070 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:12.488274 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:12.488233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:12.631761 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:12.631732 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:12.633874 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:27:12.633844 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a436569_84af_4899_aa75_37b6a6134b32.slice/crio-2f6a29e3daf134c63442ed0e6e610fc5c8c0b306fc723639138dfcaac2285c72 WatchSource:0}: Error finding container 2f6a29e3daf134c63442ed0e6e610fc5c8c0b306fc723639138dfcaac2285c72: Status 404 returned error can't find the container with id 2f6a29e3daf134c63442ed0e6e610fc5c8c0b306fc723639138dfcaac2285c72 Jul 09 04:27:12.663543 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:12.663479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" event={"ID":"9a436569-84af-4899-aa75-37b6a6134b32","Type":"ContainerStarted","Data":"2f6a29e3daf134c63442ed0e6e610fc5c8c0b306fc723639138dfcaac2285c72"} Jul 09 04:27:13.670088 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:13.670048 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" event={"ID":"9a436569-84af-4899-aa75-37b6a6134b32","Type":"ContainerStarted","Data":"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0"} Jul 09 04:27:13.670503 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:13.670207 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:13.686158 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:13.686106 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" podStartSLOduration=3.274494619 podStartE2EDuration="3.686092261s" totalImagesPullingTime="411.597642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:27:13.684961931 +0000 UTC m=+1633.288920710" watchObservedRunningTime="2026-07-09 04:27:13.686092261 +0000 UTC m=+1633.290051002" Jul 09 04:27:19.681169 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:19.681134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:22.787236 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:22.787184 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:22.788094 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:22.788034 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" podUID="9a436569-84af-4899-aa75-37b6a6134b32" containerName="maas-api" containerID="cri-o://70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0" gracePeriod=30 Jul 09 04:27:23.052012 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.051977 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:23.112571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.112529 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca\") pod \"9a436569-84af-4899-aa75-37b6a6134b32\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " Jul 09 04:27:23.112571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.112574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") pod \"9a436569-84af-4899-aa75-37b6a6134b32\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " Jul 09 04:27:23.112812 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.112634 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m5fg\" (UniqueName: \"kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg\") pod \"9a436569-84af-4899-aa75-37b6a6134b32\" (UID: \"9a436569-84af-4899-aa75-37b6a6134b32\") " Jul 09 04:27:23.112928 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.112903 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a436569-84af-4899-aa75-37b6a6134b32" (UID: "9a436569-84af-4899-aa75-37b6a6134b32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:27:23.113016 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.113003 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a436569-84af-4899-aa75-37b6a6134b32-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:27:23.114762 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.114735 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a436569-84af-4899-aa75-37b6a6134b32" (UID: "9a436569-84af-4899-aa75-37b6a6134b32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:27:23.114859 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.114806 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg" (OuterVolumeSpecName: "kube-api-access-6m5fg") pod "9a436569-84af-4899-aa75-37b6a6134b32" (UID: "9a436569-84af-4899-aa75-37b6a6134b32"). InnerVolumeSpecName "kube-api-access-6m5fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:27:23.213870 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.213830 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a436569-84af-4899-aa75-37b6a6134b32-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:27:23.213870 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.213863 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5fg\" (UniqueName: \"kubernetes.io/projected/9a436569-84af-4899-aa75-37b6a6134b32-kube-api-access-6m5fg\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:27:23.729071 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.729037 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a436569-84af-4899-aa75-37b6a6134b32" containerID="70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0" exitCode=0 Jul 09 04:27:23.729297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.729083 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" Jul 09 04:27:23.729297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.729127 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" event={"ID":"9a436569-84af-4899-aa75-37b6a6134b32","Type":"ContainerDied","Data":"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0"} Jul 09 04:27:23.729297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.729189 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r" event={"ID":"9a436569-84af-4899-aa75-37b6a6134b32","Type":"ContainerDied","Data":"2f6a29e3daf134c63442ed0e6e610fc5c8c0b306fc723639138dfcaac2285c72"} Jul 09 04:27:23.729297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.729220 2570 scope.go:117] "RemoveContainer" containerID="70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0" Jul 09 04:27:23.739571 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.739551 2570 scope.go:117] "RemoveContainer" containerID="70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0" Jul 09 04:27:23.739871 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:23.739848 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0\": container with ID starting with 70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0 not found: ID does not exist" containerID="70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0" Jul 09 04:27:23.739952 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.739885 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0"} err="failed to get container status \"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0\": rpc error: code = NotFound desc = could not find container \"70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0\": container with ID starting with 70e0fc2e6435e8f465fe69d7e49b34dd779010e626ffbb751a4ab1bc57596bb0 not found: ID does not exist" Jul 09 04:27:23.753699 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.753662 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:23.760599 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:23.760566 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c4e5fa4-5c49dc4b4d-gjm2r"] Jul 09 04:27:24.921387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:24.921335 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a436569-84af-4899-aa75-37b6a6134b32" path="/var/lib/kubelet/pods/9a436569-84af-4899-aa75-37b6a6134b32/volumes" Jul 09 04:27:51.577822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.577783 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:27:51.578507 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.578487 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a436569-84af-4899-aa75-37b6a6134b32" containerName="maas-api" Jul 09 04:27:51.578590 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.578509 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a436569-84af-4899-aa75-37b6a6134b32" containerName="maas-api" Jul 09 04:27:51.578644 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.578628 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a436569-84af-4899-aa75-37b6a6134b32" containerName="maas-api" Jul 09 04:27:51.582329 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.582306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.584312 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.584292 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-11e0a3fa-serving-cert\"" Jul 09 04:27:51.592758 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.592734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:27:51.601892 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.601865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s5x\" (UniqueName: \"kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.602057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.601929 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.602057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.602034 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.702958 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.702921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.703157 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.703003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.703157 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.703084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s5x\" (UniqueName: \"kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.703297 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:51.703163 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-serving-cert: secret "maas-api-e2e-mt-11e0a3fa-serving-cert" not found Jul 09 04:27:51.703297 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:27:51.703240 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls podName:c7d13f88-b545-41d4-af99-0fd9074d88c3 nodeName:}" failed. No retries permitted until 2026-07-09 04:27:52.203213143 +0000 UTC m=+1671.807171879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls") pod "maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" (UID: "c7d13f88-b545-41d4-af99-0fd9074d88c3") : secret "maas-api-e2e-mt-11e0a3fa-serving-cert" not found Jul 09 04:27:51.703749 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.703722 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:51.710925 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:51.710900 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s5x\" (UniqueName: \"kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:52.207504 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:52.207468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:52.209862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:52.209837 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") pod \"maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:52.494579 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:52.494495 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:52.841549 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:52.841517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:27:52.843557 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:27:52.843530 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d13f88_b545_41d4_af99_0fd9074d88c3.slice/crio-d5e609d9ccbaa14f860dad54cdf91900114aebdb039f85af19f49ef9363b7775 WatchSource:0}: Error finding container d5e609d9ccbaa14f860dad54cdf91900114aebdb039f85af19f49ef9363b7775: Status 404 returned error can't find the container with id d5e609d9ccbaa14f860dad54cdf91900114aebdb039f85af19f49ef9363b7775 Jul 09 04:27:52.889156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:52.889121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" event={"ID":"c7d13f88-b545-41d4-af99-0fd9074d88c3","Type":"ContainerStarted","Data":"d5e609d9ccbaa14f860dad54cdf91900114aebdb039f85af19f49ef9363b7775"} Jul 09 04:27:53.895814 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:53.895777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" event={"ID":"c7d13f88-b545-41d4-af99-0fd9074d88c3","Type":"ContainerStarted","Data":"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9"} Jul 09 04:27:53.896216 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:53.895883 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:27:53.914702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:53.914644 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" podStartSLOduration=2.520855897 podStartE2EDuration="2.914627749s" totalImagesPullingTime="393.771852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:27:53.912526763 +0000 UTC m=+1673.516485504" watchObservedRunningTime="2026-07-09 04:27:53.914627749 +0000 UTC m=+1673.518586489" Jul 09 04:27:59.906605 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:27:59.906571 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:28:29.108457 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.108340 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:28:29.109080 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.108581 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" podUID="c7d13f88-b545-41d4-af99-0fd9074d88c3" containerName="maas-api" containerID="cri-o://a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9" gracePeriod=30 Jul 09 04:28:29.369215 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.369149 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:28:29.486387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.486335 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75s5x\" (UniqueName: \"kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x\") pod \"c7d13f88-b545-41d4-af99-0fd9074d88c3\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " Jul 09 04:28:29.486387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.486395 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca\") pod \"c7d13f88-b545-41d4-af99-0fd9074d88c3\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " Jul 09 04:28:29.486665 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.486539 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") pod \"c7d13f88-b545-41d4-af99-0fd9074d88c3\" (UID: \"c7d13f88-b545-41d4-af99-0fd9074d88c3\") " Jul 09 04:28:29.486869 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.486833 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7d13f88-b545-41d4-af99-0fd9074d88c3" (UID: "c7d13f88-b545-41d4-af99-0fd9074d88c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:28:29.488706 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.488670 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7d13f88-b545-41d4-af99-0fd9074d88c3" (UID: "c7d13f88-b545-41d4-af99-0fd9074d88c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:28:29.488822 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.488725 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x" (OuterVolumeSpecName: "kube-api-access-75s5x") pod "c7d13f88-b545-41d4-af99-0fd9074d88c3" (UID: "c7d13f88-b545-41d4-af99-0fd9074d88c3"). InnerVolumeSpecName "kube-api-access-75s5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:28:29.588247 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.588207 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d13f88-b545-41d4-af99-0fd9074d88c3-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:28:29.588247 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.588249 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s5x\" (UniqueName: \"kubernetes.io/projected/c7d13f88-b545-41d4-af99-0fd9074d88c3-kube-api-access-75s5x\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:28:29.588471 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:29.588263 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d13f88-b545-41d4-af99-0fd9074d88c3-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:28:30.093932 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.093895 2570 generic.go:356] "Generic (PLEG): container finished" podID="c7d13f88-b545-41d4-af99-0fd9074d88c3" containerID="a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9" exitCode=0 Jul 09 04:28:30.094118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.093946 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" Jul 09 04:28:30.094118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.093969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" event={"ID":"c7d13f88-b545-41d4-af99-0fd9074d88c3","Type":"ContainerDied","Data":"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9"} Jul 09 04:28:30.094118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.094023 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6" event={"ID":"c7d13f88-b545-41d4-af99-0fd9074d88c3","Type":"ContainerDied","Data":"d5e609d9ccbaa14f860dad54cdf91900114aebdb039f85af19f49ef9363b7775"} Jul 09 04:28:30.094118 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.094053 2570 scope.go:117] "RemoveContainer" containerID="a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9" Jul 09 04:28:30.103293 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.103268 2570 scope.go:117] "RemoveContainer" containerID="a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9" Jul 09 04:28:30.103597 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:28:30.103572 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9\": container with ID starting with a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9 not found: ID does not exist" containerID="a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9" Jul 09 04:28:30.103694 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.103603 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9"} err="failed to get container status \"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9\": rpc error: code = NotFound desc = could not find container \"a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9\": container with ID starting with a823ef6fa1fefc4902aa86ce9c3f6c5a3fa248f520c81d914a807b83ad61a6f9 not found: ID does not exist" Jul 09 04:28:30.115358 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.115322 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:28:30.119492 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.119469 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11e0a3fa-745978c86c-qffx6"] Jul 09 04:28:30.923011 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:28:30.922979 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d13f88-b545-41d4-af99-0fd9074d88c3" path="/var/lib/kubelet/pods/c7d13f88-b545-41d4-af99-0fd9074d88c3/volumes" Jul 09 04:30:03.181318 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:03.181213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:30:03.194737 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:03.191016 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:30:20.291021 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.290982 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:30:20.291739 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.291709 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d13f88-b545-41d4-af99-0fd9074d88c3" containerName="maas-api" Jul 09 04:30:20.291739 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.291733 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d13f88-b545-41d4-af99-0fd9074d88c3" containerName="maas-api" Jul 09 04:30:20.291952 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.291834 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d13f88-b545-41d4-af99-0fd9074d88c3" containerName="maas-api" Jul 09 04:30:20.294679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.294659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.296679 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.296659 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7aa53b-serving-cert\"" Jul 09 04:30:20.304067 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.304041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:30:20.421988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.421951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.422178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.422038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2mb\" (UniqueName: \"kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.422178 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.422067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.523274 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.523237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2mb\" (UniqueName: \"kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.523484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.523282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.523484 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.523392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.523484 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:30:20.523465 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-serving-cert: secret "maas-api-e2e-inf-a-7aa53b-serving-cert" not found Jul 09 04:30:20.523667 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:30:20.523546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls podName:f5f13223-f5d0-41e4-a0cd-c7c480316947 nodeName:}" failed. No retries permitted until 2026-07-09 04:30:21.023524086 +0000 UTC m=+1820.627482805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls") pod "maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" (UID: "f5f13223-f5d0-41e4-a0cd-c7c480316947") : secret "maas-api-e2e-inf-a-7aa53b-serving-cert" not found Jul 09 04:30:20.523996 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.523977 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:20.532274 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:20.532253 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2mb\" (UniqueName: \"kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:21.028988 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.028950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:21.031549 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.031517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") pod \"maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:21.207246 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.207209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:21.353707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.353676 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:30:21.355425 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:30:21.355398 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f13223_f5d0_41e4_a0cd_c7c480316947.slice/crio-143de140010858299d6265e3be77430166309d7c8712ec51b2f2e1bdc0d2c8a2 WatchSource:0}: Error finding container 143de140010858299d6265e3be77430166309d7c8712ec51b2f2e1bdc0d2c8a2: Status 404 returned error can't find the container with id 143de140010858299d6265e3be77430166309d7c8712ec51b2f2e1bdc0d2c8a2 Jul 09 04:30:21.357213 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.357192 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:30:21.681228 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:21.681142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" event={"ID":"f5f13223-f5d0-41e4-a0cd-c7c480316947","Type":"ContainerStarted","Data":"143de140010858299d6265e3be77430166309d7c8712ec51b2f2e1bdc0d2c8a2"} Jul 09 04:30:22.688175 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:22.688136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" event={"ID":"f5f13223-f5d0-41e4-a0cd-c7c480316947","Type":"ContainerStarted","Data":"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec"} Jul 09 04:30:22.688585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:22.688257 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:22.703999 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:22.703937 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" podStartSLOduration=2.2429894949999998 podStartE2EDuration="2.703918398s" totalImagesPullingTime="460.928903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:22.703724599 +0000 UTC m=+1822.307683373" watchObservedRunningTime="2026-07-09 04:30:22.703918398 +0000 UTC m=+1822.307877139" Jul 09 04:30:28.461878 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.461841 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:30:28.465650 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.465626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.467573 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.467552 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a259cf-serving-cert\"" Jul 09 04:30:28.474472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.474448 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:30:28.604558 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.604527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.604738 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.604586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcznx\" (UniqueName: \"kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.604738 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.604624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.700101 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.700070 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:30:28.705551 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.705527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.705677 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.705574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcznx\" (UniqueName: \"kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.705677 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.705603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.706145 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.706124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.707927 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.707907 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.714261 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.714207 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcznx\" (UniqueName: \"kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx\") pod \"maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.778229 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.778198 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:28.918256 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:30:28.918229 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aca3e74_0ec2_407c_b50f_6d822aca6111.slice/crio-0beb480a00340c7b30be1fcdc4c4329360bad742613d113b09c4ae23dece9590 WatchSource:0}: Error finding container 0beb480a00340c7b30be1fcdc4c4329360bad742613d113b09c4ae23dece9590: Status 404 returned error can't find the container with id 0beb480a00340c7b30be1fcdc4c4329360bad742613d113b09c4ae23dece9590 Jul 09 04:30:28.922590 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:28.922568 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:30:29.733221 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:29.733177 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" event={"ID":"2aca3e74-0ec2-407c-b50f-6d822aca6111","Type":"ContainerStarted","Data":"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93"} Jul 09 04:30:29.733684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:29.733246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" event={"ID":"2aca3e74-0ec2-407c-b50f-6d822aca6111","Type":"ContainerStarted","Data":"0beb480a00340c7b30be1fcdc4c4329360bad742613d113b09c4ae23dece9590"} Jul 09 04:30:29.733684 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:29.733274 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:30:29.749297 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:29.749253 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" podStartSLOduration=1.4452612120000001 podStartE2EDuration="1.749238977s" totalImagesPullingTime="303.977765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:29.747503903 +0000 UTC m=+1829.351462646" watchObservedRunningTime="2026-07-09 04:30:29.749238977 +0000 UTC m=+1829.353197718" Jul 09 04:30:35.744495 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:30:35.744457 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:31:43.420903 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.420523 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:31:43.420903 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.420780 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" podUID="f5f13223-f5d0-41e4-a0cd-c7c480316947" containerName="maas-api" containerID="cri-o://a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec" gracePeriod=30 Jul 09 04:31:43.680524 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.680414 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:31:43.832789 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.832759 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") pod \"f5f13223-f5d0-41e4-a0cd-c7c480316947\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " Jul 09 04:31:43.832979 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.832959 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca\") pod \"f5f13223-f5d0-41e4-a0cd-c7c480316947\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " Jul 09 04:31:43.833039 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.832997 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2mb\" (UniqueName: \"kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb\") pod \"f5f13223-f5d0-41e4-a0cd-c7c480316947\" (UID: \"f5f13223-f5d0-41e4-a0cd-c7c480316947\") " Jul 09 04:31:43.833330 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.833303 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5f13223-f5d0-41e4-a0cd-c7c480316947" (UID: "f5f13223-f5d0-41e4-a0cd-c7c480316947"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:31:43.834860 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.834835 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5f13223-f5d0-41e4-a0cd-c7c480316947" (UID: "f5f13223-f5d0-41e4-a0cd-c7c480316947"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:31:43.834992 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.834975 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb" (OuterVolumeSpecName: "kube-api-access-2m2mb") pod "f5f13223-f5d0-41e4-a0cd-c7c480316947" (UID: "f5f13223-f5d0-41e4-a0cd-c7c480316947"). InnerVolumeSpecName "kube-api-access-2m2mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:31:43.934950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.934860 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f13223-f5d0-41e4-a0cd-c7c480316947-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:31:43.934950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.934894 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f13223-f5d0-41e4-a0cd-c7c480316947-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:31:43.934950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:43.934905 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2mb\" (UniqueName: \"kubernetes.io/projected/f5f13223-f5d0-41e4-a0cd-c7c480316947-kube-api-access-2m2mb\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:31:44.130694 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.130657 2570 generic.go:356] "Generic (PLEG): container finished" podID="f5f13223-f5d0-41e4-a0cd-c7c480316947" containerID="a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec" exitCode=0 Jul 09 04:31:44.130861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.130705 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" Jul 09 04:31:44.130861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.130729 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" event={"ID":"f5f13223-f5d0-41e4-a0cd-c7c480316947","Type":"ContainerDied","Data":"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec"} Jul 09 04:31:44.130861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.130772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c" event={"ID":"f5f13223-f5d0-41e4-a0cd-c7c480316947","Type":"ContainerDied","Data":"143de140010858299d6265e3be77430166309d7c8712ec51b2f2e1bdc0d2c8a2"} Jul 09 04:31:44.130861 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.130795 2570 scope.go:117] "RemoveContainer" containerID="a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec" Jul 09 04:31:44.140660 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.140644 2570 scope.go:117] "RemoveContainer" containerID="a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec" Jul 09 04:31:44.140912 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:31:44.140895 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec\": container with ID starting with a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec not found: ID does not exist" containerID="a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec" Jul 09 04:31:44.140978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.140919 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec"} err="failed to get container status \"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec\": rpc error: code = NotFound desc = could not find container \"a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec\": container with ID starting with a9b72a05d875abb37ce3359cbd20ac41627a127d6806311a6888386706e2c6ec not found: ID does not exist" Jul 09 04:31:44.151514 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.151492 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:31:44.157639 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.157614 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7aa53b-75ccc4468b-tl85c"] Jul 09 04:31:44.922778 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:31:44.922743 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f13223-f5d0-41e4-a0cd-c7c480316947" path="/var/lib/kubelet/pods/f5f13223-f5d0-41e4-a0cd-c7c480316947/volumes" Jul 09 04:32:26.449509 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.449473 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:32:26.450103 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.449731 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" podUID="2aca3e74-0ec2-407c-b50f-6d822aca6111" containerName="maas-api" containerID="cri-o://bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93" gracePeriod=30 Jul 09 04:32:26.717442 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.717413 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:32:26.850658 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.850619 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls\") pod \"2aca3e74-0ec2-407c-b50f-6d822aca6111\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " Jul 09 04:32:26.850843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.850729 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcznx\" (UniqueName: \"kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx\") pod \"2aca3e74-0ec2-407c-b50f-6d822aca6111\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " Jul 09 04:32:26.850843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.850750 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca\") pod \"2aca3e74-0ec2-407c-b50f-6d822aca6111\" (UID: \"2aca3e74-0ec2-407c-b50f-6d822aca6111\") " Jul 09 04:32:26.851204 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.851179 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2aca3e74-0ec2-407c-b50f-6d822aca6111" (UID: "2aca3e74-0ec2-407c-b50f-6d822aca6111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:32:26.852825 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.852797 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2aca3e74-0ec2-407c-b50f-6d822aca6111" (UID: "2aca3e74-0ec2-407c-b50f-6d822aca6111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:32:26.852904 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.852845 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx" (OuterVolumeSpecName: "kube-api-access-vcznx") pod "2aca3e74-0ec2-407c-b50f-6d822aca6111" (UID: "2aca3e74-0ec2-407c-b50f-6d822aca6111"). InnerVolumeSpecName "kube-api-access-vcznx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:32:26.952137 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.952052 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcznx\" (UniqueName: \"kubernetes.io/projected/2aca3e74-0ec2-407c-b50f-6d822aca6111-kube-api-access-vcznx\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:26.952137 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.952083 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aca3e74-0ec2-407c-b50f-6d822aca6111-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:26.952137 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:26.952095 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aca3e74-0ec2-407c-b50f-6d822aca6111-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:27.367151 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.367114 2570 generic.go:356] "Generic (PLEG): container finished" podID="2aca3e74-0ec2-407c-b50f-6d822aca6111" containerID="bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93" exitCode=0 Jul 09 04:32:27.367413 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.367157 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" Jul 09 04:32:27.367413 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.367198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" event={"ID":"2aca3e74-0ec2-407c-b50f-6d822aca6111","Type":"ContainerDied","Data":"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93"} Jul 09 04:32:27.367413 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.367256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g" event={"ID":"2aca3e74-0ec2-407c-b50f-6d822aca6111","Type":"ContainerDied","Data":"0beb480a00340c7b30be1fcdc4c4329360bad742613d113b09c4ae23dece9590"} Jul 09 04:32:27.367413 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.367279 2570 scope.go:117] "RemoveContainer" containerID="bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93" Jul 09 04:32:27.378975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.378941 2570 scope.go:117] "RemoveContainer" containerID="bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93" Jul 09 04:32:27.379377 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:32:27.379335 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93\": container with ID starting with bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93 not found: ID does not exist" containerID="bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93" Jul 09 04:32:27.379460 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.379380 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93"} err="failed to get container status \"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93\": rpc error: code = NotFound desc = could not find container \"bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93\": container with ID starting with bec2e0221a44260aa8d4cda650d1b677d8fd826e65f21a6b0ba024e0b6e5ab93 not found: ID does not exist" Jul 09 04:32:27.382862 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.382822 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:32:27.386513 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:27.386485 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a259cf-5f686fb97d-r9m8g"] Jul 09 04:32:28.921776 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:28.921741 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aca3e74-0ec2-407c-b50f-6d822aca6111" path="/var/lib/kubelet/pods/2aca3e74-0ec2-407c-b50f-6d822aca6111/volumes" Jul 09 04:32:36.937741 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.937695 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:36.938436 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938418 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aca3e74-0ec2-407c-b50f-6d822aca6111" containerName="maas-api" Jul 09 04:32:36.938497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938439 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aca3e74-0ec2-407c-b50f-6d822aca6111" containerName="maas-api" Jul 09 04:32:36.938497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938457 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f13223-f5d0-41e4-a0cd-c7c480316947" containerName="maas-api" Jul 09 04:32:36.938497 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938467 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f13223-f5d0-41e4-a0cd-c7c480316947" containerName="maas-api" Jul 09 04:32:36.938595 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938576 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aca3e74-0ec2-407c-b50f-6d822aca6111" containerName="maas-api" Jul 09 04:32:36.938629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.938592 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f13223-f5d0-41e4-a0cd-c7c480316947" containerName="maas-api" Jul 09 04:32:36.942232 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.942210 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:36.944390 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.944333 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ed5c7e-serving-cert\"" Jul 09 04:32:36.949871 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:36.949837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:37.045524 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.045474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85jh\" (UniqueName: \"kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.045728 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.045584 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.045728 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.045616 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.146470 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.146421 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85jh\" (UniqueName: \"kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.146661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.146591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.146661 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.146643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.146753 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:32:37.146733 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-serving-cert: secret "maas-api-e2e-api-a-ed5c7e-serving-cert" not found Jul 09 04:32:37.146820 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:32:37.146807 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls podName:0f7762c7-ed0e-4978-9676-4864b666d268 nodeName:}" failed. No retries permitted until 2026-07-09 04:32:37.646791762 +0000 UTC m=+1957.250750480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls") pod "maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" (UID: "0f7762c7-ed0e-4978-9676-4864b666d268") : secret "maas-api-e2e-api-a-ed5c7e-serving-cert" not found Jul 09 04:32:37.147240 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.147215 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.157386 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.157323 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85jh\" (UniqueName: \"kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.652251 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.652204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.654779 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.654757 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") pod \"maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.855113 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.855074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:37.995055 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:37.995021 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:37.996797 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:32:37.996769 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7762c7_ed0e_4978_9676_4864b666d268.slice/crio-a01eaf3583dfe920f96dffd4fea42670d1e1c54534135bb8eca088df62e167b7 WatchSource:0}: Error finding container a01eaf3583dfe920f96dffd4fea42670d1e1c54534135bb8eca088df62e167b7: Status 404 returned error can't find the container with id a01eaf3583dfe920f96dffd4fea42670d1e1c54534135bb8eca088df62e167b7 Jul 09 04:32:38.431533 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:38.431499 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" event={"ID":"0f7762c7-ed0e-4978-9676-4864b666d268","Type":"ContainerStarted","Data":"a01eaf3583dfe920f96dffd4fea42670d1e1c54534135bb8eca088df62e167b7"} Jul 09 04:32:39.439828 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:39.439794 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" event={"ID":"0f7762c7-ed0e-4978-9676-4864b666d268","Type":"ContainerStarted","Data":"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a"} Jul 09 04:32:39.440274 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:39.439859 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:39.456772 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:39.456725 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" podStartSLOduration=3.030408002 podStartE2EDuration="3.456710752s" totalImagesPullingTime="426.30275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:32:39.45477155 +0000 UTC m=+1959.058730291" watchObservedRunningTime="2026-07-09 04:32:39.456710752 +0000 UTC m=+1959.060669493" Jul 09 04:32:44.993625 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:44.993589 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:32:44.997739 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:44.997716 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.000677 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.000654 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a54d19-serving-cert\"" Jul 09 04:32:45.007416 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.007391 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:32:45.022550 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.022515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.022704 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.022685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpcb\" (UniqueName: \"kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.022778 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.022765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.123953 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.123917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpcb\" (UniqueName: \"kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.124112 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.123982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.124112 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.124058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.124683 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.124665 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.126493 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.126467 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.132023 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.132001 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpcb\" (UniqueName: \"kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb\") pod \"maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.311049 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.311018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:45.452093 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.452063 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:45.458373 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.458309 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:32:45.460613 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:32:45.460586 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea8df46_2d59_4557_84d5_d9685242fa83.slice/crio-a65f7e554e242256d09cd634fe87bcfae4cb2af9d76272e5ab9989be912363aa WatchSource:0}: Error finding container a65f7e554e242256d09cd634fe87bcfae4cb2af9d76272e5ab9989be912363aa: Status 404 returned error can't find the container with id a65f7e554e242256d09cd634fe87bcfae4cb2af9d76272e5ab9989be912363aa Jul 09 04:32:45.478252 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:45.478209 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" event={"ID":"7ea8df46-2d59-4557-84d5-d9685242fa83","Type":"ContainerStarted","Data":"a65f7e554e242256d09cd634fe87bcfae4cb2af9d76272e5ab9989be912363aa"} Jul 09 04:32:46.487199 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:46.487165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" event={"ID":"7ea8df46-2d59-4557-84d5-d9685242fa83","Type":"ContainerStarted","Data":"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d"} Jul 09 04:32:46.487594 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:46.487241 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:46.504399 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:46.504331 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" podStartSLOduration=2.143839346 podStartE2EDuration="2.504315591s" totalImagesPullingTime="360.476245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:32:46.502073997 +0000 UTC m=+1966.106032752" watchObservedRunningTime="2026-07-09 04:32:46.504315591 +0000 UTC m=+1966.108274332" Jul 09 04:32:52.498156 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:52.498124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:32:58.255630 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.255538 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:58.256164 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.255789 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" podUID="0f7762c7-ed0e-4978-9676-4864b666d268" containerName="maas-api" containerID="cri-o://ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a" gracePeriod=30 Jul 09 04:32:58.526796 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.526769 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:58.554514 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.554482 2570 generic.go:356] "Generic (PLEG): container finished" podID="0f7762c7-ed0e-4978-9676-4864b666d268" containerID="ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a" exitCode=0 Jul 09 04:32:58.554702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.554531 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" Jul 09 04:32:58.554702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.554562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" event={"ID":"0f7762c7-ed0e-4978-9676-4864b666d268","Type":"ContainerDied","Data":"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a"} Jul 09 04:32:58.554702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.554619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj" event={"ID":"0f7762c7-ed0e-4978-9676-4864b666d268","Type":"ContainerDied","Data":"a01eaf3583dfe920f96dffd4fea42670d1e1c54534135bb8eca088df62e167b7"} Jul 09 04:32:58.554702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.554659 2570 scope.go:117] "RemoveContainer" containerID="ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a" Jul 09 04:32:58.564108 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.564084 2570 scope.go:117] "RemoveContainer" containerID="ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a" Jul 09 04:32:58.564491 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:32:58.564465 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a\": container with ID starting with ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a not found: ID does not exist" containerID="ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a" Jul 09 04:32:58.564595 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.564502 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a"} err="failed to get container status \"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a\": rpc error: code = NotFound desc = could not find container \"ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a\": container with ID starting with ed4ec88ba2654cf261771176e7ba6bfd0e75136617d1cfd82584891bc675a90a not found: ID does not exist" Jul 09 04:32:58.672029 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.671989 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca\") pod \"0f7762c7-ed0e-4978-9676-4864b666d268\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " Jul 09 04:32:58.672218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.672087 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") pod \"0f7762c7-ed0e-4978-9676-4864b666d268\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " Jul 09 04:32:58.672218 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.672172 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l85jh\" (UniqueName: \"kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh\") pod \"0f7762c7-ed0e-4978-9676-4864b666d268\" (UID: \"0f7762c7-ed0e-4978-9676-4864b666d268\") " Jul 09 04:32:58.672389 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.672363 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f7762c7-ed0e-4978-9676-4864b666d268" (UID: "0f7762c7-ed0e-4978-9676-4864b666d268"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:32:58.672617 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.672589 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f7762c7-ed0e-4978-9676-4864b666d268-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:58.674163 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.674140 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f7762c7-ed0e-4978-9676-4864b666d268" (UID: "0f7762c7-ed0e-4978-9676-4864b666d268"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:32:58.674301 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.674279 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh" (OuterVolumeSpecName: "kube-api-access-l85jh") pod "0f7762c7-ed0e-4978-9676-4864b666d268" (UID: "0f7762c7-ed0e-4978-9676-4864b666d268"). InnerVolumeSpecName "kube-api-access-l85jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:32:58.774171 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.774068 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f7762c7-ed0e-4978-9676-4864b666d268-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:58.774171 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.774114 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85jh\" (UniqueName: \"kubernetes.io/projected/0f7762c7-ed0e-4978-9676-4864b666d268-kube-api-access-l85jh\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:32:58.876007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.875971 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:58.880182 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.880150 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed5c7e-75bf85b479-pqmwj"] Jul 09 04:32:58.922450 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:32:58.922416 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f7762c7-ed0e-4978-9676-4864b666d268" path="/var/lib/kubelet/pods/0f7762c7-ed0e-4978-9676-4864b666d268/volumes" Jul 09 04:33:05.060472 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.060436 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:33:05.061075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.060657 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" podUID="7ea8df46-2d59-4557-84d5-d9685242fa83" containerName="maas-api" containerID="cri-o://fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d" gracePeriod=30 Jul 09 04:33:05.321671 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.321599 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:33:05.449975 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.449926 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca\") pod \"7ea8df46-2d59-4557-84d5-d9685242fa83\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " Jul 09 04:33:05.450160 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.449994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls\") pod \"7ea8df46-2d59-4557-84d5-d9685242fa83\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " Jul 09 04:33:05.450160 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.450097 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpcb\" (UniqueName: \"kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb\") pod \"7ea8df46-2d59-4557-84d5-d9685242fa83\" (UID: \"7ea8df46-2d59-4557-84d5-d9685242fa83\") " Jul 09 04:33:05.450295 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.450271 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ea8df46-2d59-4557-84d5-d9685242fa83" (UID: "7ea8df46-2d59-4557-84d5-d9685242fa83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:33:05.450410 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.450394 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea8df46-2d59-4557-84d5-d9685242fa83-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:33:05.452428 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.452402 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ea8df46-2d59-4557-84d5-d9685242fa83" (UID: "7ea8df46-2d59-4557-84d5-d9685242fa83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:33:05.452540 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.452516 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb" (OuterVolumeSpecName: "kube-api-access-pgpcb") pod "7ea8df46-2d59-4557-84d5-d9685242fa83" (UID: "7ea8df46-2d59-4557-84d5-d9685242fa83"). InnerVolumeSpecName "kube-api-access-pgpcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:33:05.551291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.551255 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpcb\" (UniqueName: \"kubernetes.io/projected/7ea8df46-2d59-4557-84d5-d9685242fa83-kube-api-access-pgpcb\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:33:05.551291 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.551288 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea8df46-2d59-4557-84d5-d9685242fa83-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:33:05.595100 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.595009 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ea8df46-2d59-4557-84d5-d9685242fa83" containerID="fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d" exitCode=0 Jul 09 04:33:05.595100 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.595052 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" Jul 09 04:33:05.595100 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.595077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" event={"ID":"7ea8df46-2d59-4557-84d5-d9685242fa83","Type":"ContainerDied","Data":"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d"} Jul 09 04:33:05.595331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.595120 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k" event={"ID":"7ea8df46-2d59-4557-84d5-d9685242fa83","Type":"ContainerDied","Data":"a65f7e554e242256d09cd634fe87bcfae4cb2af9d76272e5ab9989be912363aa"} Jul 09 04:33:05.595331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.595141 2570 scope.go:117] "RemoveContainer" containerID="fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d" Jul 09 04:33:05.604550 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.604529 2570 scope.go:117] "RemoveContainer" containerID="fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d" Jul 09 04:33:05.604820 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:33:05.604801 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d\": container with ID starting with fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d not found: ID does not exist" containerID="fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d" Jul 09 04:33:05.604917 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.604830 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d"} err="failed to get container status \"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d\": rpc error: code = NotFound desc = could not find container \"fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d\": container with ID starting with fb4077e50ce5fc5780014efa9ed56bcac9299a1c3e7187b843c27e1a0bc1a39d not found: ID does not exist" Jul 09 04:33:05.615498 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.615470 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:33:05.619283 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:05.619262 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a54d19-6b67b8dc5c-8hh6k"] Jul 09 04:33:06.921220 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:06.921189 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea8df46-2d59-4557-84d5-d9685242fa83" path="/var/lib/kubelet/pods/7ea8df46-2d59-4557-84d5-d9685242fa83/volumes" Jul 09 04:33:15.164666 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.164620 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:33:15.165393 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165369 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea8df46-2d59-4557-84d5-d9685242fa83" containerName="maas-api" Jul 09 04:33:15.165393 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165394 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea8df46-2d59-4557-84d5-d9685242fa83" containerName="maas-api" Jul 09 04:33:15.165532 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165431 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7762c7-ed0e-4978-9676-4864b666d268" containerName="maas-api" Jul 09 04:33:15.165532 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165440 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7762c7-ed0e-4978-9676-4864b666d268" containerName="maas-api" Jul 09 04:33:15.165596 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165550 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7762c7-ed0e-4978-9676-4864b666d268" containerName="maas-api" Jul 09 04:33:15.165596 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.165564 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea8df46-2d59-4557-84d5-d9685242fa83" containerName="maas-api" Jul 09 04:33:15.170713 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.170692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.173144 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.173114 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-48a065-serving-cert\"" Jul 09 04:33:15.183680 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.183651 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:33:15.245466 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.245430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.245634 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.245470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.245634 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.245553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxv4\" (UniqueName: \"kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.346677 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.346640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.346894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.346683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.346894 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.346748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxv4\" (UniqueName: \"kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.346894 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:33:15.346797 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-serving-cert: secret "maas-api-e2e-shared-a-48a065-serving-cert" not found Jul 09 04:33:15.346894 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:33:15.346881 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls podName:284634e6-f252-413a-926d-679f2a5db54f nodeName:}" failed. No retries permitted until 2026-07-09 04:33:15.846862943 +0000 UTC m=+1995.450821662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls") pod "maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" (UID: "284634e6-f252-413a-926d-679f2a5db54f") : secret "maas-api-e2e-shared-a-48a065-serving-cert" not found Jul 09 04:33:15.347374 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.347339 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.358009 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.357981 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxv4\" (UniqueName: \"kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.850518 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.850471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:15.853019 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:15.852997 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48a065-585dbbd575-86pl8\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:16.084573 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:16.084525 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:16.225675 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:16.225591 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:33:16.228212 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:33:16.228181 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284634e6_f252_413a_926d_679f2a5db54f.slice/crio-7547863355a72ca629c2e030241d699f212a4c7009fa5f3f439b11d3e5984e13 WatchSource:0}: Error finding container 7547863355a72ca629c2e030241d699f212a4c7009fa5f3f439b11d3e5984e13: Status 404 returned error can't find the container with id 7547863355a72ca629c2e030241d699f212a4c7009fa5f3f439b11d3e5984e13 Jul 09 04:33:16.662807 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:16.662762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" event={"ID":"284634e6-f252-413a-926d-679f2a5db54f","Type":"ContainerStarted","Data":"7547863355a72ca629c2e030241d699f212a4c7009fa5f3f439b11d3e5984e13"} Jul 09 04:33:17.669797 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:17.669750 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" event={"ID":"284634e6-f252-413a-926d-679f2a5db54f","Type":"ContainerStarted","Data":"2761633329d7ed05007ef71f21ddc6c7f73777b065258f990d83dc6004df0018"} Jul 09 04:33:17.670132 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:17.669830 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:17.688307 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:17.688252 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" podStartSLOduration=2.343234465 podStartE2EDuration="2.68823012s" totalImagesPullingTime="344.995655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:33:17.684902798 +0000 UTC m=+1997.288861565" watchObservedRunningTime="2026-07-09 04:33:17.68823012 +0000 UTC m=+1997.292188862" Jul 09 04:33:23.271408 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.271364 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:33:23.276154 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.276127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.278331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.278306 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-332f6a-serving-cert\"" Jul 09 04:33:23.284830 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.284798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:33:23.328603 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.328566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.328806 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.328627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.328806 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.328686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmp6\" (UniqueName: \"kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.429382 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.429332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.429575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.429405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.429575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.429436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmp6\" (UniqueName: \"kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.429575 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:33:23.429496 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-serving-cert: secret "maas-api-e2e-shared-b-332f6a-serving-cert" not found Jul 09 04:33:23.429575 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:33:23.429568 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls podName:196b07e0-b12e-449a-8e61-58ce3a0443bd nodeName:}" failed. No retries permitted until 2026-07-09 04:33:23.929546177 +0000 UTC m=+2003.533504914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls") pod "maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" (UID: "196b07e0-b12e-449a-8e61-58ce3a0443bd") : secret "maas-api-e2e-shared-b-332f6a-serving-cert" not found Jul 09 04:33:23.430108 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.430086 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.440057 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.440013 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmp6\" (UniqueName: \"kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.681102 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.681068 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:33:23.935248 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.935124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:23.937729 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:23.937692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:24.190812 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:24.190719 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:24.339446 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:24.339385 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:33:24.342002 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:33:24.341965 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196b07e0_b12e_449a_8e61_58ce3a0443bd.slice/crio-b95a39085c94ab0c811f3814504613f2108546704a7aab91e4c7baeb7bae16fa WatchSource:0}: Error finding container b95a39085c94ab0c811f3814504613f2108546704a7aab91e4c7baeb7bae16fa: Status 404 returned error can't find the container with id b95a39085c94ab0c811f3814504613f2108546704a7aab91e4c7baeb7bae16fa Jul 09 04:33:24.710555 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:24.710512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" event={"ID":"196b07e0-b12e-449a-8e61-58ce3a0443bd","Type":"ContainerStarted","Data":"b95a39085c94ab0c811f3814504613f2108546704a7aab91e4c7baeb7bae16fa"} Jul 09 04:33:25.717644 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:25.717604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" event={"ID":"196b07e0-b12e-449a-8e61-58ce3a0443bd","Type":"ContainerStarted","Data":"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba"} Jul 09 04:33:25.718053 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:25.717698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:33:25.734503 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:25.734449 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" podStartSLOduration=2.247426032 podStartE2EDuration="2.734430378s" totalImagesPullingTime="487.004346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:33:25.732096856 +0000 UTC m=+2005.336055620" watchObservedRunningTime="2026-07-09 04:33:25.734430378 +0000 UTC m=+2005.338389119" Jul 09 04:33:31.728282 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:33:31.728248 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:35:03.216168 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:35:03.216046 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:35:03.225750 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:35:03.225726 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:36:33.506239 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.506193 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:36:33.506856 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.506522 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" podUID="284634e6-f252-413a-926d-679f2a5db54f" containerName="maas-api" containerID="cri-o://2761633329d7ed05007ef71f21ddc6c7f73777b065258f990d83dc6004df0018" gracePeriod=30 Jul 09 04:36:33.675331 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.675289 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" podUID="284634e6-f252-413a-926d-679f2a5db54f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused" Jul 09 04:36:33.740467 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.740434 2570 generic.go:356] "Generic (PLEG): container finished" podID="284634e6-f252-413a-926d-679f2a5db54f" containerID="2761633329d7ed05007ef71f21ddc6c7f73777b065258f990d83dc6004df0018" exitCode=0 Jul 09 04:36:33.740637 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.740504 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" event={"ID":"284634e6-f252-413a-926d-679f2a5db54f","Type":"ContainerDied","Data":"2761633329d7ed05007ef71f21ddc6c7f73777b065258f990d83dc6004df0018"} Jul 09 04:36:33.769307 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.769233 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:36:33.925377 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.925315 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkxv4\" (UniqueName: \"kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4\") pod \"284634e6-f252-413a-926d-679f2a5db54f\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " Jul 09 04:36:33.925555 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.925392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca\") pod \"284634e6-f252-413a-926d-679f2a5db54f\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " Jul 09 04:36:33.925555 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.925438 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") pod \"284634e6-f252-413a-926d-679f2a5db54f\" (UID: \"284634e6-f252-413a-926d-679f2a5db54f\") " Jul 09 04:36:33.925812 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.925789 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "284634e6-f252-413a-926d-679f2a5db54f" (UID: "284634e6-f252-413a-926d-679f2a5db54f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:36:33.927587 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.927567 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "284634e6-f252-413a-926d-679f2a5db54f" (UID: "284634e6-f252-413a-926d-679f2a5db54f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:36:33.927707 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:33.927612 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4" (OuterVolumeSpecName: "kube-api-access-wkxv4") pod "284634e6-f252-413a-926d-679f2a5db54f" (UID: "284634e6-f252-413a-926d-679f2a5db54f"). InnerVolumeSpecName "kube-api-access-wkxv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:34.027243 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.027147 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxv4\" (UniqueName: \"kubernetes.io/projected/284634e6-f252-413a-926d-679f2a5db54f-kube-api-access-wkxv4\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:34.027243 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.027193 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284634e6-f252-413a-926d-679f2a5db54f-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:34.027485 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.027282 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284634e6-f252-413a-926d-679f2a5db54f-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:34.747006 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.746976 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" Jul 09 04:36:34.747494 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.746976 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8" event={"ID":"284634e6-f252-413a-926d-679f2a5db54f","Type":"ContainerDied","Data":"7547863355a72ca629c2e030241d699f212a4c7009fa5f3f439b11d3e5984e13"} Jul 09 04:36:34.747494 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.747105 2570 scope.go:117] "RemoveContainer" containerID="2761633329d7ed05007ef71f21ddc6c7f73777b065258f990d83dc6004df0018" Jul 09 04:36:34.773590 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.773558 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:36:34.782743 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.782717 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48a065-585dbbd575-86pl8"] Jul 09 04:36:34.922266 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:34.922230 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="284634e6-f252-413a-926d-679f2a5db54f" path="/var/lib/kubelet/pods/284634e6-f252-413a-926d-679f2a5db54f/volumes" Jul 09 04:36:40.326413 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.326377 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:36:40.326831 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.326600 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" podUID="196b07e0-b12e-449a-8e61-58ce3a0443bd" containerName="maas-api" containerID="cri-o://54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba" gracePeriod=30 Jul 09 04:36:40.584714 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.584587 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:36:40.695860 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.695823 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") pod \"196b07e0-b12e-449a-8e61-58ce3a0443bd\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " Jul 09 04:36:40.696032 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.695908 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvmp6\" (UniqueName: \"kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6\") pod \"196b07e0-b12e-449a-8e61-58ce3a0443bd\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " Jul 09 04:36:40.696032 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.696001 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca\") pod \"196b07e0-b12e-449a-8e61-58ce3a0443bd\" (UID: \"196b07e0-b12e-449a-8e61-58ce3a0443bd\") " Jul 09 04:36:40.696418 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.696394 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "196b07e0-b12e-449a-8e61-58ce3a0443bd" (UID: "196b07e0-b12e-449a-8e61-58ce3a0443bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:36:40.697994 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.697970 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6" (OuterVolumeSpecName: "kube-api-access-jvmp6") pod "196b07e0-b12e-449a-8e61-58ce3a0443bd" (UID: "196b07e0-b12e-449a-8e61-58ce3a0443bd"). InnerVolumeSpecName "kube-api-access-jvmp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:40.698075 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.698011 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "196b07e0-b12e-449a-8e61-58ce3a0443bd" (UID: "196b07e0-b12e-449a-8e61-58ce3a0443bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:36:40.783223 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.783186 2570 generic.go:356] "Generic (PLEG): container finished" podID="196b07e0-b12e-449a-8e61-58ce3a0443bd" containerID="54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba" exitCode=0 Jul 09 04:36:40.783421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.783235 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" Jul 09 04:36:40.783421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.783237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" event={"ID":"196b07e0-b12e-449a-8e61-58ce3a0443bd","Type":"ContainerDied","Data":"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba"} Jul 09 04:36:40.783421 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.783401 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88" event={"ID":"196b07e0-b12e-449a-8e61-58ce3a0443bd","Type":"ContainerDied","Data":"b95a39085c94ab0c811f3814504613f2108546704a7aab91e4c7baeb7bae16fa"} Jul 09 04:36:40.783581 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.783440 2570 scope.go:117] "RemoveContainer" containerID="54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba" Jul 09 04:36:40.793753 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.793733 2570 scope.go:117] "RemoveContainer" containerID="54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba" Jul 09 04:36:40.794064 ip-10-0-137-40 kubenswrapper[2570]: E0709 04:36:40.794045 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba\": container with ID starting with 54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba not found: ID does not exist" containerID="54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba" Jul 09 04:36:40.794125 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.794077 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba"} err="failed to get container status \"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba\": rpc error: code = NotFound desc = could not find container \"54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba\": container with ID starting with 54aa897680f47c401f87395e18dceb5a0c8c03766a48484b76cd2032fe1478ba not found: ID does not exist" Jul 09 04:36:40.797187 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.797161 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmp6\" (UniqueName: \"kubernetes.io/projected/196b07e0-b12e-449a-8e61-58ce3a0443bd-kube-api-access-jvmp6\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.797187 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.797187 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196b07e0-b12e-449a-8e61-58ce3a0443bd-openshift-service-ca\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.797320 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.797197 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196b07e0-b12e-449a-8e61-58ce3a0443bd-maas-api-tls\") on node \"ip-10-0-137-40.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.804327 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.804296 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:36:40.808093 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.808066 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-332f6a-7847bf8895-4kk88"] Jul 09 04:36:40.921751 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:40.921663 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196b07e0-b12e-449a-8e61-58ce3a0443bd" path="/var/lib/kubelet/pods/196b07e0-b12e-449a-8e61-58ce3a0443bd/volumes" Jul 09 04:36:51.535479 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:51.535435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7r6lt_88f92474-496f-4511-98a9-e6a68bd39469/manager/0.log" Jul 09 04:36:51.662933 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:51.662904 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r24tj_ad26ce3b-e76a-4f79-a763-70dfce342f0c/manager/0.log" Jul 09 04:36:52.040843 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:52.040811 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-txdhh_b11e1980-9db0-41c8-b7f9-bbf072fc3fe5/manager/0.log" Jul 09 04:36:52.156618 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:52.156591 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-lr7gw_01ea5194-86e0-4f90-bbc8-8402e3414c4c/copy-manifests/0.log" Jul 09 04:36:52.162368 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:52.162327 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-lr7gw_01ea5194-86e0-4f90-bbc8-8402e3414c4c/manager/0.log" Jul 09 04:36:53.922610 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:53.922578 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tbv8w_7effc24d-0ee0-4142-9541-7fca9157d238/manager/0.log" Jul 09 04:36:54.041188 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:54.041147 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5p5kn_f0da4996-4c44-44ca-999d-a53306a2b310/manager/0.log" Jul 09 04:36:54.288070 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:54.287988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hflmz_22caa450-d143-48fe-92a8-c55b91e3a3e1/registry-server/0.log" Jul 09 04:36:54.442989 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:54.442956 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hd2mz_2aa85859-28fd-4299-a303-f2a0228c1b59/manager/0.log" Jul 09 04:36:54.686403 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:54.686372 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gsshw_f35e121d-d88e-4d7f-97f2-013ac20547d3/manager/0.log" Jul 09 04:36:55.289634 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:55.289596 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-nzt2t_4d3f3d68-b055-4244-a04b-371c52da3e19/kube-auth-proxy/0.log" Jul 09 04:36:55.539264 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:55.539234 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/1.log" Jul 09 04:36:55.665121 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:55.665090 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ts96l_3a713f60-4870-4735-a23c-cee19409ccbf/payload-pre-processing/0.log" Jul 09 04:36:55.913284 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:55.913257 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59497797c7-88275_f89e374d-dac0-4a80-9412-f0e7a94d8a82/router/0.log" Jul 09 04:36:56.273602 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:56.273573 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2cvf9_a6b7e496-af03-446f-8da4-1c494e82030d/main/0.log" Jul 09 04:36:56.509325 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:56.509294 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-dph76_bb33c918-775b-47a5-ad21-5ba4526057e4/main/0.log" Jul 09 04:36:56.630197 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:36:56.630168 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs2mzj_f8f1bfe5-d85e-4c5f-9ea9-66196812f3da/main/0.log" Jul 09 04:37:04.057800 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:04.057760 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-54pxr_21f4e347-8cf3-4ecc-b6c1-18e9c590e662/global-pull-secret-syncer/0.log" Jul 09 04:37:04.215365 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:04.215321 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bbtkq_15e21662-7592-4c77-80ef-1f8b029160fc/konnectivity-agent/0.log" Jul 09 04:37:04.283336 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:04.283306 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-40.ec2.internal_2e6fcc3f92e7e2909dcce9ec4feecf63/haproxy/0.log" Jul 09 04:37:08.917973 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:08.917927 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tbv8w_7effc24d-0ee0-4142-9541-7fca9157d238/manager/0.log" Jul 09 04:37:08.949276 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:08.949243 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5p5kn_f0da4996-4c44-44ca-999d-a53306a2b310/manager/0.log" Jul 09 04:37:09.015584 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:09.015540 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hflmz_22caa450-d143-48fe-92a8-c55b91e3a3e1/registry-server/0.log" Jul 09 04:37:09.194246 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:09.194145 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hd2mz_2aa85859-28fd-4299-a303-f2a0228c1b59/manager/0.log" Jul 09 04:37:09.304006 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:09.303972 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gsshw_f35e121d-d88e-4d7f-97f2-013ac20547d3/manager/0.log" Jul 09 04:37:11.090298 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.090266 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-7d4qd_a0e1b35d-5a64-4eaa-a85c-9899af48c4a6/cluster-monitoring-operator/0.log" Jul 09 04:37:11.195016 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.194983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f7d77f95f-2phch_fad11bd7-6357-4ae2-8966-8684d2ad24b7/metrics-server/0.log" Jul 09 04:37:11.403061 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.402968 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mn96m_8de4c8f0-c912-43fb-8542-f884c6f58243/node-exporter/0.log" Jul 09 04:37:11.420664 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.420635 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mn96m_8de4c8f0-c912-43fb-8542-f884c6f58243/kube-rbac-proxy/0.log" Jul 09 04:37:11.438712 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.438688 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mn96m_8de4c8f0-c912-43fb-8542-f884c6f58243/init-textfile/0.log" Jul 09 04:37:11.544085 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.544057 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/prometheus/0.log" Jul 09 04:37:11.570055 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.570021 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/config-reloader/0.log" Jul 09 04:37:11.594846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.594824 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/thanos-sidecar/0.log" Jul 09 04:37:11.610038 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.610011 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/kube-rbac-proxy-web/0.log" Jul 09 04:37:11.631544 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.631511 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/kube-rbac-proxy/0.log" Jul 09 04:37:11.658820 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.658730 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/kube-rbac-proxy-thanos/0.log" Jul 09 04:37:11.674771 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.674751 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_82150a79-ab36-4b9e-803d-6e2ea4edd657/init-config-reloader/0.log" Jul 09 04:37:11.767609 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.767582 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-7fdlm_7091ee9b-53d4-4196-828a-c44000073418/prometheus-operator-admission-webhook/0.log" Jul 09 04:37:11.876022 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.875985 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/thanos-query/0.log" Jul 09 04:37:11.893557 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.893529 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/kube-rbac-proxy-web/0.log" Jul 09 04:37:11.912572 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.912493 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/kube-rbac-proxy/0.log" Jul 09 04:37:11.929007 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.928981 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/prom-label-proxy/0.log" Jul 09 04:37:11.945985 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.945957 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/kube-rbac-proxy-rules/0.log" Jul 09 04:37:11.962627 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:11.962590 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77b5c685d9-gn84z_0b891226-48ff-4e86-84bd-610185b43eed/kube-rbac-proxy-metrics/0.log" Jul 09 04:37:13.286538 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:13.286477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-pkbg8_e878bddb-eee6-4794-90be-2db53f9ad7cc/networking-console-plugin/0.log" Jul 09 04:37:13.899816 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:13.899784 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/1.log" Jul 09 04:37:13.907529 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:13.907493 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-jh6p7_419714ad-2c57-4966-8ed2-173135fac77d/console-operator/2.log" Jul 09 04:37:14.403090 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.403058 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84dd999c45-knggw_56208214-d183-45e8-bb36-b42398fbbd43/console/0.log" Jul 09 04:37:14.682101 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682008 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8"] Jul 09 04:37:14.682585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682566 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="284634e6-f252-413a-926d-679f2a5db54f" containerName="maas-api" Jul 09 04:37:14.682585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682588 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="284634e6-f252-413a-926d-679f2a5db54f" containerName="maas-api" Jul 09 04:37:14.682764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682620 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196b07e0-b12e-449a-8e61-58ce3a0443bd" containerName="maas-api" Jul 09 04:37:14.682764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682629 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="196b07e0-b12e-449a-8e61-58ce3a0443bd" containerName="maas-api" Jul 09 04:37:14.682764 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682756 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="196b07e0-b12e-449a-8e61-58ce3a0443bd" containerName="maas-api" Jul 09 04:37:14.682911 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.682774 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="284634e6-f252-413a-926d-679f2a5db54f" containerName="maas-api" Jul 09 04:37:14.685987 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.685967 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.687885 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.687864 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85fmr\"/\"openshift-service-ca.crt\"" Jul 09 04:37:14.687978 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.687903 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-85fmr\"/\"default-dockercfg-hslpv\"" Jul 09 04:37:14.688448 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.688435 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85fmr\"/\"kube-root-ca.crt\"" Jul 09 04:37:14.694846 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.694823 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8"] Jul 09 04:37:14.838437 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.838398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-sys\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.838437 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.838435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nps\" (UniqueName: \"kubernetes.io/projected/20ea6e5e-d40e-4297-b465-310c6a891c41-kube-api-access-p2nps\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.838649 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.838555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-proc\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.838649 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.838586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-podres\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.838725 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.838653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-lib-modules\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-sys\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939387 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nps\" (UniqueName: \"kubernetes.io/projected/20ea6e5e-d40e-4297-b465-310c6a891c41-kube-api-access-p2nps\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-proc\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-sys\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-podres\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-lib-modules\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-proc\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939575 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939541 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-podres\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.939790 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.939612 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20ea6e5e-d40e-4297-b465-310c6a891c41-lib-modules\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.946916 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.946885 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nps\" (UniqueName: \"kubernetes.io/projected/20ea6e5e-d40e-4297-b465-310c6a891c41-kube-api-access-p2nps\") pod \"perf-node-gather-daemonset-7jlx8\" (UID: \"20ea6e5e-d40e-4297-b465-310c6a891c41\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:14.996501 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:14.996466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:15.073400 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:15.073339 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-fzjms_df0f4a8d-f57a-4be0-b7f6-30ed0d655ec9/volume-data-source-validator/0.log" Jul 09 04:37:15.136070 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:15.136043 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8"] Jul 09 04:37:15.136881 ip-10-0-137-40 kubenswrapper[2570]: W0709 04:37:15.136853 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20ea6e5e_d40e_4297_b465_310c6a891c41.slice/crio-859d9eb70b2c3682b500fe8e3b1dfa5cb12ff42b993c8de1f2165983db46f81d WatchSource:0}: Error finding container 859d9eb70b2c3682b500fe8e3b1dfa5cb12ff42b993c8de1f2165983db46f81d: Status 404 returned error can't find the container with id 859d9eb70b2c3682b500fe8e3b1dfa5cb12ff42b993c8de1f2165983db46f81d Jul 09 04:37:15.977666 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:15.977624 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" event={"ID":"20ea6e5e-d40e-4297-b465-310c6a891c41","Type":"ContainerStarted","Data":"a8623d3adda32f5e80c974041fcd88458715ce9263adf8951cee7c9db47af5be"} Jul 09 04:37:15.978043 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:15.977688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" event={"ID":"20ea6e5e-d40e-4297-b465-310c6a891c41","Type":"ContainerStarted","Data":"859d9eb70b2c3682b500fe8e3b1dfa5cb12ff42b993c8de1f2165983db46f81d"} Jul 09 04:37:15.978043 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:15.977724 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:16.106851 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:16.106816 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5s6xd_94ba1740-99fb-47a0-9c83-0689c076cc34/dns/0.log" Jul 09 04:37:16.127631 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:16.127604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5s6xd_94ba1740-99fb-47a0-9c83-0689c076cc34/kube-rbac-proxy/0.log" Jul 09 04:37:16.188987 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:16.188958 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-78sl6_d4bae520-6f0d-47e6-b75c-950c05499b3c/dns-node-resolver/0.log" Jul 09 04:37:16.802213 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:16.802179 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-djx94_81c5e01e-0e76-493e-9a70-9125d05cd963/node-ca/0.log" Jul 09 04:37:17.874174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:17.874149 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-nzt2t_4d3f3d68-b055-4244-a04b-371c52da3e19/kube-auth-proxy/0.log" Jul 09 04:37:17.917530 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:17.917489 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/0.log" Jul 09 04:37:17.972728 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:17.972693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/1.log" Jul 09 04:37:18.023292 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:18.023259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ts96l_3a713f60-4870-4735-a23c-cee19409ccbf/payload-pre-processing/0.log" Jul 09 04:37:18.124762 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:18.124681 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59497797c7-88275_f89e374d-dac0-4a80-9412-f0e7a94d8a82/router/0.log" Jul 09 04:37:18.811174 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:18.811138 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggd2l_f47ad52f-c4b5-4416-b63b-3e2ae739712a/serve-healthcheck-canary/0.log" Jul 09 04:37:19.243451 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:19.243345 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wc4zl_ce0cf1d9-9b4a-4dae-9333-0973ae5928d3/insights-operator/0.log" Jul 09 04:37:19.244397 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:19.244376 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wc4zl_ce0cf1d9-9b4a-4dae-9333-0973ae5928d3/insights-operator/1.log" Jul 09 04:37:19.334277 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:19.334241 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pppkm_b2c5f642-6ed5-442b-8e2c-e74ee36237bc/kube-rbac-proxy/0.log" Jul 09 04:37:19.388489 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:19.388458 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pppkm_b2c5f642-6ed5-442b-8e2c-e74ee36237bc/exporter/0.log" Jul 09 04:37:19.437950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:19.437919 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pppkm_b2c5f642-6ed5-442b-8e2c-e74ee36237bc/extractor/0.log" Jul 09 04:37:21.595136 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:21.595102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7r6lt_88f92474-496f-4511-98a9-e6a68bd39469/manager/0.log" Jul 09 04:37:21.781934 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:21.781903 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r24tj_ad26ce3b-e76a-4f79-a763-70dfce342f0c/manager/0.log" Jul 09 04:37:21.992943 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:21.992860 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-txdhh_b11e1980-9db0-41c8-b7f9-bbf072fc3fe5/manager/0.log" Jul 09 04:37:21.996585 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:21.996559 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" Jul 09 04:37:22.013334 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:22.013289 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-7jlx8" podStartSLOduration=8.013273155 podStartE2EDuration="8.013273155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:37:15.992830788 +0000 UTC m=+2235.596789529" watchObservedRunningTime="2026-07-09 04:37:22.013273155 +0000 UTC m=+2241.617231896" Jul 09 04:37:22.017503 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:22.017477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-lr7gw_01ea5194-86e0-4f90-bbc8-8402e3414c4c/manager/0.log" Jul 09 04:37:22.033742 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:22.033716 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-lr7gw_01ea5194-86e0-4f90-bbc8-8402e3414c4c/copy-manifests/0.log" Jul 09 04:37:24.690864 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:24.690825 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-nzt2t_4d3f3d68-b055-4244-a04b-371c52da3e19/kube-auth-proxy/0.log" Jul 09 04:37:24.724162 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:24.724137 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/0.log" Jul 09 04:37:24.784569 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:24.784475 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/1.log" Jul 09 04:37:24.829717 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:24.829688 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ts96l_3a713f60-4870-4735-a23c-cee19409ccbf/payload-pre-processing/0.log" Jul 09 04:37:24.908283 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:24.908254 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59497797c7-88275_f89e374d-dac0-4a80-9412-f0e7a94d8a82/router/0.log" Jul 09 04:37:26.215799 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:26.215770 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-nzt2t_4d3f3d68-b055-4244-a04b-371c52da3e19/kube-auth-proxy/0.log" Jul 09 04:37:26.250629 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:26.250604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/0.log" Jul 09 04:37:26.305992 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:26.305962 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-tmvfs_a607c53e-79b0-4406-b0f6-c7bc9a7ddc2c/istio-proxy/1.log" Jul 09 04:37:26.353458 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:26.353424 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ts96l_3a713f60-4870-4735-a23c-cee19409ccbf/payload-pre-processing/0.log" Jul 09 04:37:26.437726 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:26.437699 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59497797c7-88275_f89e374d-dac0-4a80-9412-f0e7a94d8a82/router/0.log" Jul 09 04:37:28.791950 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:28.791895 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-k7n5c_14d79e88-ae22-4f3c-8414-fbaca11408b2/kube-storage-version-migrator-operator/1.log" Jul 09 04:37:28.793432 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:28.793403 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-k7n5c_14d79e88-ae22-4f3c-8414-fbaca11408b2/kube-storage-version-migrator-operator/0.log" Jul 09 04:37:29.837828 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.837796 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/kube-multus-additional-cni-plugins/0.log" Jul 09 04:37:29.857250 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.857213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/egress-router-binary-copy/0.log" Jul 09 04:37:29.873702 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.873676 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/cni-plugins/0.log" Jul 09 04:37:29.889087 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.889066 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/bond-cni-plugin/0.log" Jul 09 04:37:29.916323 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.916301 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/routeoverride-cni/0.log" Jul 09 04:37:29.931745 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.931723 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/whereabouts-cni-bincopy/0.log" Jul 09 04:37:29.947332 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:29.947311 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6p5qz_e9787e2f-3fde-4bb3-8d4d-a058c3b46757/whereabouts-cni/0.log" Jul 09 04:37:30.314016 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:30.313987 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n5d5b_cf60a3fd-3717-4311-ba96-b89248760807/kube-multus/0.log" Jul 09 04:37:30.432051 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:30.432017 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xp2dp_2fd95163-b562-4bd6-9157-c69ebc28e76b/network-metrics-daemon/0.log" Jul 09 04:37:30.444244 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:30.444218 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xp2dp_2fd95163-b562-4bd6-9157-c69ebc28e76b/kube-rbac-proxy/0.log" Jul 09 04:37:31.746548 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.746504 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/ovn-controller/0.log" Jul 09 04:37:31.767553 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.767527 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/ovn-acl-logging/0.log" Jul 09 04:37:31.787827 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.787801 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/kube-rbac-proxy-node/0.log" Jul 09 04:37:31.805345 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.805298 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 04:37:31.818832 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.818808 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/northd/0.log" Jul 09 04:37:31.834767 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.834741 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/nbdb/0.log" Jul 09 04:37:31.850672 ip-10-0-137-40 kubenswrapper[2570]: I0709 04:37:31.850628 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qqqng_9be4d8b2-5274-40cf-b789-f2992ceb9004/sbdb/0.log"