Jul 15 08:00:22.451805 ip-10-0-139-41 systemd[1]: Starting Kubernetes Kubelet... Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.903621 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912044 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912061 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912065 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912068 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912071 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912075 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912078 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.920539 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912081 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912084 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912087 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912090 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912093 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912097 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912099 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912102 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912104 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912107 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912110 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912113 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912115 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912118 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912120 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912123 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912125 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912128 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912130 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912133 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.928513 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912136 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912138 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912141 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912144 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912147 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912152 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912156 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912159 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912162 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912165 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912167 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912170 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912173 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912175 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912178 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912180 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912183 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912185 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912188 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912190 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.931207 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912193 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912196 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912198 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912201 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912203 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912206 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912208 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912211 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912213 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912217 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912221 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912225 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912229 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912232 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912235 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912237 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912241 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912244 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912246 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912249 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.940902 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912252 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912254 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912257 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912259 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912262 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912265 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912267 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912270 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912272 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912275 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912277 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912280 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912283 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912285 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912288 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912290 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912294 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912296 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912299 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.943385 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912303 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912308 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912312 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912316 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912318 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912322 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912325 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912328 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912331 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912335 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912337 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912340 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.912343 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913049 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913056 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913058 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913061 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913064 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913067 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.949480 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913070 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913072 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913075 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913077 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913080 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913083 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913086 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913088 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913091 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913094 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913096 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913099 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913102 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913105 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913108 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913110 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913113 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913115 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913118 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.953150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913120 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913123 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913125 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913128 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913131 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913133 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913136 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913138 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913141 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913143 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913146 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913148 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913151 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913154 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913156 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913158 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913161 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913163 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913166 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913168 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.957769 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913171 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913173 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913176 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913178 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913181 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913185 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913188 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913190 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913193 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913195 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913198 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913200 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913204 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913209 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913214 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913218 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913221 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913224 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913227 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913230 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.965094 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913233 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913236 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913238 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913241 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913243 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913246 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913248 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913251 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913253 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913256 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913258 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913261 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913263 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913266 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913269 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913272 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913275 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913278 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913281 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913286 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.965687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913289 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913292 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913295 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913298 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913300 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913303 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913306 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913309 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913312 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913314 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913316 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913319 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913322 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913324 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913731 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913747 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913750 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913753 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913756 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.966176 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913758 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913761 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913763 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913768 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913771 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913774 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913777 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913780 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913786 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913788 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913791 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913794 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913797 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913800 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913803 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913805 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913808 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913810 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913813 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.966651 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913816 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913818 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913821 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913823 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913825 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913828 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913830 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913833 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913836 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913839 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913841 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913844 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913846 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913849 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913851 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913854 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913856 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913859 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913861 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.967129 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913864 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913866 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913871 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913873 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913876 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913880 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913883 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913886 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913889 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913891 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913894 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913897 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913899 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913902 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913904 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913907 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913909 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913912 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913914 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913917 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.967625 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913919 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913922 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913924 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913927 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913929 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913932 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913934 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913937 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913940 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913943 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913947 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913950 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913952 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913956 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913960 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913963 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913965 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913970 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913973 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913976 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.968150 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913979 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913981 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913984 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913986 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913989 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913991 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913994 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913996 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.913998 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914001 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914004 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914006 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914009 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914011 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914014 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914017 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914083 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914090 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914095 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914102 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914108 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 08:00:22.968685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914113 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914119 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914125 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914131 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914134 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914140 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914143 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914146 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914150 2565 flags.go:64] FLAG: --cgroup-root="" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914153 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914156 2565 flags.go:64] FLAG: --client-ca-file="" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914160 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914164 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914169 2565 flags.go:64] FLAG: --cluster-domain="" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914172 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914175 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914178 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914182 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914186 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914189 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914192 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914195 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914198 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914201 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 08:00:22.969171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914204 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914207 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914210 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914214 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914217 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914220 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914223 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914226 2565 flags.go:64] FLAG: --enable-server="true" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914229 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914233 2565 flags.go:64] FLAG: --event-burst="100" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914236 2565 flags.go:64] FLAG: --event-qps="50" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914238 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914241 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914246 2565 flags.go:64] FLAG: --eviction-hard="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914253 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914256 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914260 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914263 2565 flags.go:64] FLAG: --eviction-soft="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914266 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914269 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914273 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914276 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914279 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914281 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914284 2565 flags.go:64] FLAG: --feature-gates="" Jul 15 08:00:22.969794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914288 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914291 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914294 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914297 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914300 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914303 2565 flags.go:64] FLAG: --help="false" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914306 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-41.ec2.internal" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914309 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914312 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914314 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914318 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914321 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914324 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914327 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914329 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914332 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914335 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914338 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914341 2565 flags.go:64] FLAG: --kube-reserved="" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914344 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914348 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914351 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914354 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914358 2565 flags.go:64] FLAG: --lock-file="" Jul 15 08:00:22.970410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914361 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914364 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914367 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914390 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914394 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914410 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914413 2565 flags.go:64] FLAG: --logging-format="text" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914417 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914420 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914423 2565 flags.go:64] FLAG: --manifest-url="" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914426 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914431 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914434 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914438 2565 flags.go:64] FLAG: --max-pods="110" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914441 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914444 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914447 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914450 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914453 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914456 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914459 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914467 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914470 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914473 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 08:00:22.970980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914476 2565 flags.go:64] FLAG: --pod-cidr="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914479 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914484 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914486 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914490 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914494 2565 flags.go:64] FLAG: --port="10250" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914497 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914501 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e78a79051338d0f" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914504 2565 flags.go:64] FLAG: --qos-reserved="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914507 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914510 2565 flags.go:64] FLAG: --register-node="true" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914513 2565 flags.go:64] FLAG: --register-with-taints="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914517 2565 flags.go:64] FLAG: --registry-burst="10" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914520 2565 flags.go:64] FLAG: --registry-qps="5" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914523 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914525 2565 flags.go:64] FLAG: --reserved-memory="" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914529 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914532 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914536 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914539 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914541 2565 flags.go:64] FLAG: --runonce="false" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914544 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914547 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914550 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914553 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 08:00:22.971624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914556 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914559 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914562 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914565 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914568 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914571 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914574 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914577 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914580 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914582 2565 flags.go:64] FLAG: --system-cgroups="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914585 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914590 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914595 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914597 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914603 2565 flags.go:64] FLAG: --tls-min-version="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914606 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914609 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914612 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914614 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914618 2565 flags.go:64] FLAG: --v="2" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914622 2565 flags.go:64] FLAG: --version="false" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914626 2565 flags.go:64] FLAG: --vmodule="" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914630 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.914634 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914752 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.972222 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914755 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914759 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914762 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914765 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914768 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914770 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914773 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914775 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914778 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914780 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914783 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914785 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914788 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914791 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914793 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914796 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914798 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914801 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914803 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914808 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.972864 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914810 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914814 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914817 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914819 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914823 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914826 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914828 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914831 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914833 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914837 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914839 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914842 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914844 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914847 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914849 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914852 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914856 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914860 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914863 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.973378 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914866 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914869 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914871 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914874 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914877 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914879 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914882 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914884 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914887 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914889 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914892 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914894 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914898 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914901 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914905 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914907 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914910 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914912 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914916 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.973855 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914920 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914923 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914926 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914928 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914932 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914934 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914937 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914940 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914942 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914945 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914947 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914951 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914955 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914958 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914960 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914963 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914965 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914968 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914970 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.974307 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914973 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914975 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914978 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914980 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914983 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914985 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914990 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914992 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914996 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.914999 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915001 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915004 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915006 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915009 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915012 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915014 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915016 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915019 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915022 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915025 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.974787 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.915027 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.916814 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.924866 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.924886 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.925896 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.925914 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.925997 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926004 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926009 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926014 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926018 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926023 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926028 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926033 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926037 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.975280 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926042 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926046 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926049 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926056 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926061 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926066 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926071 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926075 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926080 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926085 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926090 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926093 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926097 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926101 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926106 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926111 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926115 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926120 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926123 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926128 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.975756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926132 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926137 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926141 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926144 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926148 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926152 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926156 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926160 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926164 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926169 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926173 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926177 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926181 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926185 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926189 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926193 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926197 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926201 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926205 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926208 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.976244 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926215 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926220 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926224 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926229 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926233 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926237 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926244 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926249 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926254 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926258 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926262 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926266 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926270 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926274 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926278 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926282 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926286 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926290 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926294 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926298 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.976727 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926302 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926306 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926310 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926314 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926318 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926322 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926325 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926329 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926333 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926337 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926341 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926345 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926349 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926353 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926357 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926361 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926365 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926368 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926374 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.977379 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926378 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926382 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926386 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926390 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926394 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926417 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926422 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926426 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926430 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926434 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926440 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.926448 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926606 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926614 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 08:00:22.977977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926618 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926623 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926627 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926630 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926634 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926638 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926642 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926646 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926649 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926654 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926657 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926661 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926665 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926669 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926673 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926677 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926681 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926686 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926690 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926694 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 08:00:22.978440 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926698 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926702 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926706 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926711 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926714 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926718 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926721 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926725 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926729 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926733 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926737 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926741 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926745 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926748 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926752 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926756 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926760 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926763 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926767 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 08:00:22.978919 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926771 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926776 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926781 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926784 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926788 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926792 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926796 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926799 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926803 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926809 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926816 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926820 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926825 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926830 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926835 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926839 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926843 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926847 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926851 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 08:00:22.979376 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926855 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926859 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926863 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926866 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926873 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926877 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926881 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926885 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926888 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926892 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926896 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926900 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926904 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926908 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926913 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926917 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926921 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926925 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926929 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926933 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 08:00:22.980027 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926937 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926940 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926944 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926950 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926954 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926957 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926961 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926967 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926971 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926976 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926980 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926985 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926989 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926993 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.926997 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.927001 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.927006 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.927010 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 08:00:22.980578 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:22.927014 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.927021 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.927161 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.929927 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.931726 2565 server.go:1019] "Starting client certificate rotation" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.931833 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.931863 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.934877 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.935990 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.948425 2565 log.go:25] "Validated CRI v1 runtime API" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.951201 2565 log.go:25] "Validated CRI v1 image API" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.952095 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 08:00:22.981036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.957540 2565 fs.go:135] Filesystem UUIDs: map[08f738a2-44f2-4e13-b52b-d0ebc895a35f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c6cb817d-f5b2-44ea-8baf-3f9eb10cf690:/dev/nvme0n1p4] Jul 15 08:00:22.981335 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.957556 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.963515 2565 manager.go:216] Machine: {Timestamp:2026-07-15 08:00:22.962093433 +0000 UTC m=+0.381838227 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102650 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c187296f50db4fa0b01224a8473ff SystemUUID:ec2c1872-96f5-0db4-fa0b-01224a8473ff BootID:76b83264-4b54-4c24-9628-5b058fb801af Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:63:02:d8:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:63:02:d8:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:80:22:33:7d:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.963619 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.963696 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.967157 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.967184 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.968173 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.968185 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 08:00:22.981365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.968193 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.969897 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.971879 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.975769 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976550 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976679 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976713 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976725 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976744 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976757 2565 kubelet.go:398] "Adding apiserver pod source" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.976769 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.977208 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.977227 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 08:00:22.981750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.979867 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 08:00:22.982642 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982626 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 08:00:22.982718 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982665 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 08:00:22.982848 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982835 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 08:00:22.982904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982855 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 08:00:22.982904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982865 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 08:00:22.982904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982873 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 08:00:22.982904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982885 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 08:00:22.982904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.982894 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 08:00:22.983536 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983526 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 08:00:22.983599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983538 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 08:00:22.983599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983549 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 08:00:22.983599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983564 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 08:00:22.983599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983579 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 08:00:22.983599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.983592 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 08:00:22.984535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.984524 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 08:00:22.984591 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.984540 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 08:00:22.987575 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.987548 2565 server.go:1268] "Started kubelet" Jul 15 08:00:22.987721 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.987697 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 08:00:22.988022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.987996 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 08:00:22.988104 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.988042 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 08:00:22.988388 ip-10-0-139-41 systemd[1]: Started Kubernetes Kubelet. Jul 15 08:00:22.993186 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.993066 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 08:00:22.993272 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:22.993246 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 08:00:22.993483 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.993464 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 08:00:22.993761 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:22.993741 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 08:00:22.994887 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:22.994867 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 08:00:22.997045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:22.997031 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 15 08:00:23.001522 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.000426 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37d15c5d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:22.987523545 +0000 UTC m=+0.407268339,LastTimestamp:2026-07-15 08:00:22.987523545 +0000 UTC m=+0.407268339,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.002589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.002568 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 08:00:23.002707 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.002569 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 08:00:23.002707 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.002657 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.003339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.003323 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 08:00:23.003629 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.003611 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 08:00:23.003629 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.003631 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 08:00:23.003813 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.003798 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 15 08:00:23.003868 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.003815 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 15 08:00:23.004325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004305 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 08:00:23.004325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004325 2565 factory.go:55] Registering systemd factory Jul 15 08:00:23.004439 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004334 2565 factory.go:223] Registration of the systemd container factory successfully Jul 15 08:00:23.004972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004655 2565 factory.go:153] Registering CRI-O factory Jul 15 08:00:23.004972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004672 2565 factory.go:223] Registration of the crio container factory successfully Jul 15 08:00:23.004972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004694 2565 factory.go:103] Registering Raw factory Jul 15 08:00:23.004972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.004707 2565 manager.go:1196] Started watching for new ooms in manager Jul 15 08:00:23.005216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.005067 2565 manager.go:349] Starting recovery of all containers Jul 15 08:00:23.007228 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.007169 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 08:00:23.008180 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.008153 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 08:00:23.018367 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.018352 2565 manager.go:354] Recovery completed Jul 15 08:00:23.022947 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.022931 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.024862 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.024844 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.024948 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.024874 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.024948 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.024884 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.025460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.025440 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 08:00:23.025460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.025458 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 08:00:23.025579 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.025459 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.025579 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.025476 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 15 08:00:23.026271 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.026258 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.026317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.026288 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.026317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.026298 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.027361 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.027245 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37f4f7e05 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:23.024860677 +0000 UTC m=+0.444605471,LastTimestamp:2026-07-15 08:00:23.024860677 +0000 UTC m=+0.444605471,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.027726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.027710 2565 policy_none.go:49] "None policy: Start" Jul 15 08:00:23.027774 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.027735 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 08:00:23.027774 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.027751 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 08:00:23.029859 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.029847 2565 policy_none.go:47] "Start" Jul 15 08:00:23.037454 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.037275 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37f4fc4d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:23.024878806 +0000 UTC m=+0.444623600,LastTimestamp:2026-07-15 08:00:23.024878806 +0000 UTC m=+0.444623600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.039561 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.039479 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37f4fe998 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:23.024888216 +0000 UTC m=+0.444633009,LastTimestamp:2026-07-15 08:00:23.024888216 +0000 UTC m=+0.444633009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.047880 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.047792 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-41.ec2.internal.18c267b37f4f7e05\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37f4f7e05 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:23.024860677 +0000 UTC m=+0.444605471,LastTimestamp:2026-07-15 08:00:23.026270733 +0000 UTC m=+0.446015526,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.050505 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.050486 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8pspp" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.059044 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8pspp" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.060779 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-41.ec2.internal.18c267b37f4fc4d6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18c267b37f4fc4d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-07-15 08:00:23.024878806 +0000 UTC m=+0.444623600,LastTimestamp:2026-07-15 08:00:23.026293365 +0000 UTC m=+0.446038159,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068014 2565 manager.go:337] "Starting Device Plugin manager" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.068043 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068054 2565 server.go:87] "Starting device plugin registration server" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068281 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068290 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068390 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068493 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.068547 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.069117 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 08:00:23.084500 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.069169 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.108733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.108704 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 08:00:23.109890 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.109864 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 08:00:23.109998 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.109913 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 08:00:23.109998 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.109942 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 08:00:23.109998 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.109982 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 08:00:23.112485 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.112460 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:23.168414 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.168375 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.169444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.169425 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.169554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.169461 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.169554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.169476 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.169554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.169510 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.179139 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.179094 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.179139 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.179116 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-41.ec2.internal\": node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.210203 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.210168 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"] Jul 15 08:00:23.210314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.210238 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.212066 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.212046 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.212157 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.212080 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.212157 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.212095 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.214549 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.214536 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.214704 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.214677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.214744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.214716 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.215337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215322 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.215443 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215322 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.215443 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215350 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.215443 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215362 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.215443 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215374 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.215443 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.215393 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.217708 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.217690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.217786 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.217719 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:23.218447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.218429 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:23.218535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.218460 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:23.218535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.218473 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:23.224589 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.224572 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.241208 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.241183 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.245775 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.245759 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.305069 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.305040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.305069 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.305071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.305212 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.305091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0a59e8f4957b010cc06fb503bb4835a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"b0a59e8f4957b010cc06fb503bb4835a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.324889 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.324869 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.405764 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.405834 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.405834 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.405834 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0a59e8f4957b010cc06fb503bb4835a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"b0a59e8f4957b010cc06fb503bb4835a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.405942 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0a59e8f4957b010cc06fb503bb4835a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"b0a59e8f4957b010cc06fb503bb4835a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.405942 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.405868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181f6c79728673db7ccb0cd897d643da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"181f6c79728673db7ccb0cd897d643da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.425838 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.425812 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.526697 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.526673 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.542869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.542837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.548868 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.548853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:23.627528 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.627493 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.727900 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.727871 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.828426 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.828335 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.928726 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:23.928697 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:23.930838 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.930821 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 08:00:23.930971 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.930953 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 08:00:23.988020 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.987999 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:23.989233 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:23.989209 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:24.029128 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.029105 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.055254 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.055236 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 08:00:24.056437 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.056422 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory" Jul 15 08:00:24.056513 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.056448 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 08:00:24.056513 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.056457 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID" Jul 15 08:00:24.064561 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.064535 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 07:55:22 +0000 UTC" deadline="2028-01-06 12:00:30.930114123 +0000 UTC" Jul 15 08:00:24.064561 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.064559 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12964h0m6.865558107s" Jul 15 08:00:24.129877 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.129815 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.166687 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:24.166653 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181f6c79728673db7ccb0cd897d643da.slice/crio-074fd042a7bef77c3f648540332061302afd376ddca273579b6f5b6e64a578fc WatchSource:0}: Error finding container 074fd042a7bef77c3f648540332061302afd376ddca273579b6f5b6e64a578fc: Status 404 returned error can't find the container with id 074fd042a7bef77c3f648540332061302afd376ddca273579b6f5b6e64a578fc Jul 15 08:00:24.167087 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:24.167068 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a59e8f4957b010cc06fb503bb4835a.slice/crio-d4270819d2c0eeac63843b14b6f65061b5e1495a8342bff68ce0366cc23b9945 WatchSource:0}: Error finding container d4270819d2c0eeac63843b14b6f65061b5e1495a8342bff68ce0366cc23b9945: Status 404 returned error can't find the container with id d4270819d2c0eeac63843b14b6f65061b5e1495a8342bff68ce0366cc23b9945 Jul 15 08:00:24.170713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.170698 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:00:24.230088 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.230065 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.330472 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.330448 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.430881 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.430804 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.531455 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.531429 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found" Jul 15 08:00:24.549013 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.548993 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:24.602993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.602962 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" Jul 15 08:00:24.613599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.613582 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 08:00:24.614528 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.614517 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" Jul 15 08:00:24.626855 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.626838 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 08:00:24.977558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.977532 2565 apiserver.go:52] "Watching apiserver" Jul 15 08:00:24.987734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.987710 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 08:00:24.990263 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.990236 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","openshift-multus/multus-additional-cni-plugins-sw7h4","openshift-multus/network-metrics-daemon-jh48p","openshift-network-diagnostics/network-check-target-ptzbf","openshift-ovn-kubernetes/ovnkube-node-m7n97","kube-system/global-pull-secret-syncer-f2s78","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj","openshift-cluster-node-tuning-operator/tuned-22kqk","openshift-dns/node-resolver-9jhvd","openshift-image-registry/node-ca-hmdh8","openshift-multus/multus-kgl8n","openshift-network-operator/iptables-alerter-sxggh","kube-system/konnectivity-agent-qhb8r","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"] Jul 15 08:00:24.992509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.992491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:24.994550 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.994529 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:24.994930 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.994912 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 08:00:24.995058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.995036 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r5k5p\"" Jul 15 08:00:24.995199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.995180 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:00:24.996585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.996471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 08:00:24.996585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.996498 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:24.996585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.996579 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 08:00:24.996779 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.996626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p8n4t\"" Jul 15 08:00:24.996779 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:24.996554 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:24.998655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:24.998636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.000681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.000662 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkszc\"" Jul 15 08:00:25.000857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.000842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.001036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.001017 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 08:00:25.001369 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.001344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 08:00:25.001492 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.001417 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 08:00:25.003427 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.003203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.003427 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.003342 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:25.003565 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.003515 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 08:00:25.003793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.003775 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 08:00:25.003876 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.003862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 08:00:25.004584 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.004541 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 08:00:25.004584 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.004552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 08:00:25.004584 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.004570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 08:00:25.004766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.004542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqfd4\"" Jul 15 08:00:25.004766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.004734 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 08:00:25.005489 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.005434 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:25.005571 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.005505 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:25.007848 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.007742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.009769 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.009746 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 08:00:25.010114 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.010098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.010337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.010308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hcvx2\"" Jul 15 08:00:25.010634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.010578 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 08:00:25.010634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.010627 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 08:00:25.010897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.010882 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 08:00:25.011203 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.011187 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 08:00:25.012115 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.012099 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 08:00:25.012444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.012429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.012621 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.012591 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 08:00:25.013148 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.012911 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tlf9\"" Jul 15 08:00:25.013148 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.012986 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 08:00:25.013277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.013235 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 08:00:25.014532 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.014514 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 08:00:25.015080 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rllf\"" Jul 15 08:00:25.015862 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.015954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-kubelet\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.015954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-systemd-units\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.015954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchvt\" (UniqueName: \"kubernetes.io/projected/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-kube-api-access-lchvt\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znj8\" (UniqueName: \"kubernetes.io/projected/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-kube-api-access-8znj8\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysconfig\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.015993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-sys\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-var-lib-kubelet\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lhv\" (UniqueName: \"kubernetes.io/projected/75cda433-8419-4e55-abd6-ebded8a827bf-kube-api-access-c4lhv\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-netd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-config\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-dbus\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-run\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-lib-modules\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-tmp\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-systemd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-ovn\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-kubelet-config\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.016572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-modprobe-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-etc-tuned\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-device-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.016735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.016735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-log-socket\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-env-overrides\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-kubernetes\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4k5v\" (UniqueName: \"kubernetes.io/projected/44438dfa-d58b-48f6-9002-0bbd27cf84c5-kube-api-access-c4k5v\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-socket-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-etc-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovn-node-metrics-cert\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-script-lib\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-systemd\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.016957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-hosts-file\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-tmp-dir\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.016994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-conf\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-registration-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-bin\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-sys-fs\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-slash\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-node-log\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t4j\" (UniqueName: \"kubernetes.io/projected/d65906e9-c2f0-408c-9802-45585194971c-kube-api-access-b6t4j\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.017527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-netns\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.018086 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-var-lib-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.018086 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-host\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.018086 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.018086 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.017949 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:00:25.018568 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.018521 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 08:00:25.019181 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.019023 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 08:00:25.019181 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.019037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8hg6h\"" Jul 15 08:00:25.021128 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.020969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 08:00:25.021128 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.021102 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 08:00:25.021668 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.021344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tct8x\"" Jul 15 08:00:25.041453 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.041429 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rx9jd" Jul 15 08:00:25.049444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.049422 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rx9jd" Jul 15 08:00:25.104459 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.104436 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 08:00:25.113837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.113798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"181f6c79728673db7ccb0cd897d643da","Type":"ContainerStarted","Data":"074fd042a7bef77c3f648540332061302afd376ddca273579b6f5b6e64a578fc"} Jul 15 08:00:25.114877 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.114854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"b0a59e8f4957b010cc06fb503bb4835a","Type":"ContainerStarted","Data":"d4270819d2c0eeac63843b14b6f65061b5e1495a8342bff68ce0366cc23b9945"} Jul 15 08:00:25.117730 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.117812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-env-overrides\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.117812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-multus\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.117812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.117812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-kubernetes\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-socket-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-kubernetes\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovn-node-metrics-cert\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-socket-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.117969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.117987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-socket-dir-parent\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4cff77-0eaf-4210-9839-c8a3cbe90021-host\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-systemd\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-registration-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-bin\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-systemd\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-os-release\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118205 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-registration-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.118221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118217 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-netns\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4cff77-0eaf-4210-9839-c8a3cbe90021-serviceca\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118238 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118245 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-bin\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa330738-6386-40f9-8d31-b823ef2739cc-iptables-alerter-script\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-env-overrides\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t4j\" (UniqueName: \"kubernetes.io/projected/d65906e9-c2f0-408c-9802-45585194971c-kube-api-access-b6t4j\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-var-lib-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-conf-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-host\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-var-lib-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-systemd-units\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6ms\" (UniqueName: \"kubernetes.io/projected/aa330738-6386-40f9-8d31-b823ef2739cc-kube-api-access-6b6ms\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.118743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-host\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysconfig\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-systemd-units\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-dbus\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-kubelet\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysconfig\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-run\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-lib-modules\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-systemd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-ovn\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-kubelet-config\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-run\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-dbus\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-daemon-config\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-systemd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-lib-modules\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-kubelet-config\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-os-release\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118806 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-ovn\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-script-lib\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-device-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-log-socket\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-konnectivity-ca\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.118936 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-log-socket\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.118988 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:25.618974865 +0000 UTC m=+3.038719670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.119893 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.118937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa330738-6386-40f9-8d31-b823ef2739cc-host-slash\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-device-dir\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4k5v\" (UniqueName: \"kubernetes.io/projected/44438dfa-d58b-48f6-9002-0bbd27cf84c5-kube-api-access-c4k5v\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-etc-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-system-cni-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-hosts-file\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-etc-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119158 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-tmp-dir\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-conf\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-hosts-file\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cni-binary-copy\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cnibin\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.120537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-script-lib\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-sys-fs\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-sysctl-conf\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75cda433-8419-4e55-abd6-ebded8a827bf-sys-fs\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-tmp-dir\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-slash\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-node-log\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-k8s-cni-cncf-io\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-agent-certs\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-slash\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-netns\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-node-log\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-hostroot\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119715 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-run-netns\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq28g\" (UniqueName: \"kubernetes.io/projected/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-kube-api-access-cq28g\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzmq\" (UniqueName: \"kubernetes.io/projected/1d4cff77-0eaf-4210-9839-c8a3cbe90021-kube-api-access-mwzmq\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-kubelet\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchvt\" (UniqueName: \"kubernetes.io/projected/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-kube-api-access-lchvt\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.119862 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-etc-kubernetes\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-kubelet\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znj8\" (UniqueName: \"kubernetes.io/projected/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-kube-api-access-8znj8\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.119911 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:25.619897295 +0000 UTC m=+3.039642086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.119992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-sys\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.121713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-var-lib-kubelet\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lhv\" (UniqueName: \"kubernetes.io/projected/75cda433-8419-4e55-abd6-ebded8a827bf-kube-api-access-c4lhv\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-netd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-config\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-var-lib-kubelet\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cnibin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120119 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-host-cni-netd\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-bin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-sys\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-tmp\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-multus-certs\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjqf\" (UniqueName: \"kubernetes.io/projected/eaf327ca-0c00-409e-9a4b-d808203fcdfd-kube-api-access-ggjqf\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-run-openvswitch\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-system-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-modprobe-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-etc-tuned\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65906e9-c2f0-408c-9802-45585194971c-etc-modprobe-d\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.120606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovnkube-config\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.122092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-ovn-node-metrics-cert\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.122209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-tmp\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.122589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.122469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65906e9-c2f0-408c-9802-45585194971c-etc-tuned\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.142278 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.142246 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:25.142278 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.142273 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:25.142469 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.142287 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:25.142469 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.142358 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:25.642339824 +0000 UTC m=+3.062084603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:25.143264 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.143236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znj8\" (UniqueName: \"kubernetes.io/projected/a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6-kube-api-access-8znj8\") pod \"node-resolver-9jhvd\" (UID: \"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6\") " pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.143264 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.143238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4k5v\" (UniqueName: \"kubernetes.io/projected/44438dfa-d58b-48f6-9002-0bbd27cf84c5-kube-api-access-c4k5v\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.143818 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.143798 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lhv\" (UniqueName: \"kubernetes.io/projected/75cda433-8419-4e55-abd6-ebded8a827bf-kube-api-access-c4lhv\") pod \"aws-ebs-csi-driver-node-5cgdj\" (UID: \"75cda433-8419-4e55-abd6-ebded8a827bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.144997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.144975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchvt\" (UniqueName: \"kubernetes.io/projected/b4f8670d-87f7-4d3d-af8a-10a9afd44afe-kube-api-access-lchvt\") pod \"ovnkube-node-m7n97\" (UID: \"b4f8670d-87f7-4d3d-af8a-10a9afd44afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.145063 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.145002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t4j\" (UniqueName: \"kubernetes.io/projected/d65906e9-c2f0-408c-9802-45585194971c-kube-api-access-b6t4j\") pod \"tuned-22kqk\" (UID: \"d65906e9-c2f0-408c-9802-45585194971c\") " pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.220693 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-system-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220693 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-multus\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-multus\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-socket-dir-parent\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4cff77-0eaf-4210-9839-c8a3cbe90021-host\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-system-cni-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.220899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4cff77-0eaf-4210-9839-c8a3cbe90021-host\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-socket-dir-parent\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-os-release\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-netns\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.220986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4cff77-0eaf-4210-9839-c8a3cbe90021-serviceca\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa330738-6386-40f9-8d31-b823ef2739cc-iptables-alerter-script\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-netns\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-conf-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6ms\" (UniqueName: \"kubernetes.io/projected/aa330738-6386-40f9-8d31-b823ef2739cc-kube-api-access-6b6ms\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-kubelet\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-daemon-config\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-os-release\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-konnectivity-ca\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa330738-6386-40f9-8d31-b823ef2739cc-host-slash\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-system-cni-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cni-binary-copy\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cnibin\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-os-release\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-k8s-cni-cncf-io\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-agent-certs\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4cff77-0eaf-4210-9839-c8a3cbe90021-serviceca\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-hostroot\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-conf-dir\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq28g\" (UniqueName: \"kubernetes.io/projected/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-kube-api-access-cq28g\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.221917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa330738-6386-40f9-8d31-b823ef2739cc-iptables-alerter-script\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-kubelet\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzmq\" (UniqueName: \"kubernetes.io/projected/1d4cff77-0eaf-4210-9839-c8a3cbe90021-kube-api-access-mwzmq\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cnibin\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa330738-6386-40f9-8d31-b823ef2739cc-host-slash\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-etc-kubernetes\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-system-cni-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cnibin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-bin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-multus-certs\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjqf\" (UniqueName: \"kubernetes.io/projected/eaf327ca-0c00-409e-9a4b-d808203fcdfd-kube-api-access-ggjqf\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-multus-daemon-config\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-os-release\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-hostroot\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.222646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.221971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-multus-certs\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-etc-kubernetes\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cnibin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaf327ca-0c00-409e-9a4b-d808203fcdfd-cni-binary-copy\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-run-k8s-cni-cncf-io\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaf327ca-0c00-409e-9a4b-d808203fcdfd-host-var-lib-cni-bin\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-konnectivity-ca\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.223117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.222187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.224078 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.224061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9-agent-certs\") pod \"konnectivity-agent-qhb8r\" (UID: \"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9\") " pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.229697 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.229630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjqf\" (UniqueName: \"kubernetes.io/projected/eaf327ca-0c00-409e-9a4b-d808203fcdfd-kube-api-access-ggjqf\") pod \"multus-kgl8n\" (UID: \"eaf327ca-0c00-409e-9a4b-d808203fcdfd\") " pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.229894 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.229871 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6ms\" (UniqueName: \"kubernetes.io/projected/aa330738-6386-40f9-8d31-b823ef2739cc-kube-api-access-6b6ms\") pod \"iptables-alerter-sxggh\" (UID: \"aa330738-6386-40f9-8d31-b823ef2739cc\") " pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.230059 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.230038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzmq\" (UniqueName: \"kubernetes.io/projected/1d4cff77-0eaf-4210-9839-c8a3cbe90021-kube-api-access-mwzmq\") pod \"node-ca-hmdh8\" (UID: \"1d4cff77-0eaf-4210-9839-c8a3cbe90021\") " pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.230168 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.230055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq28g\" (UniqueName: \"kubernetes.io/projected/a203fc45-5d0c-4c8c-9e32-9f6ff42960d0-kube-api-access-cq28g\") pod \"multus-additional-cni-plugins-sw7h4\" (UID: \"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0\") " pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.302929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.302896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-22kqk" Jul 15 08:00:25.310030 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.310002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jhvd" Jul 15 08:00:25.313051 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.313024 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65906e9_c2f0_408c_9802_45585194971c.slice/crio-02df9b6d2125b41bd14df7934e2ae0d59b25ea201a80257e34ac4047e035e931 WatchSource:0}: Error finding container 02df9b6d2125b41bd14df7934e2ae0d59b25ea201a80257e34ac4047e035e931: Status 404 returned error can't find the container with id 02df9b6d2125b41bd14df7934e2ae0d59b25ea201a80257e34ac4047e035e931 Jul 15 08:00:25.319425 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.319263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d9b9a1_0ca0_49b9_ae9d_d5dc4410a2c6.slice/crio-c9097537e0c7e74d01a127f96086ff29371d49fc599023956e62989f79faf04a WatchSource:0}: Error finding container c9097537e0c7e74d01a127f96086ff29371d49fc599023956e62989f79faf04a: Status 404 returned error can't find the container with id c9097537e0c7e74d01a127f96086ff29371d49fc599023956e62989f79faf04a Jul 15 08:00:25.320268 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.320249 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" Jul 15 08:00:25.329305 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.329274 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cda433_8419_4e55_abd6_ebded8a827bf.slice/crio-cca4ce325ea2bc3994de31968642d9f814810c8be3e143a5b894b05144ce9537 WatchSource:0}: Error finding container cca4ce325ea2bc3994de31968642d9f814810c8be3e143a5b894b05144ce9537: Status 404 returned error can't find the container with id cca4ce325ea2bc3994de31968642d9f814810c8be3e143a5b894b05144ce9537 Jul 15 08:00:25.333571 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.333553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:25.340241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.340153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" Jul 15 08:00:25.341947 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.341925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f8670d_87f7_4d3d_af8a_10a9afd44afe.slice/crio-35e1b0acbe7be1bbec0f01ae5a1e681f175ae0a9981403b18d57ee1fc868f359 WatchSource:0}: Error finding container 35e1b0acbe7be1bbec0f01ae5a1e681f175ae0a9981403b18d57ee1fc868f359: Status 404 returned error can't find the container with id 35e1b0acbe7be1bbec0f01ae5a1e681f175ae0a9981403b18d57ee1fc868f359 Jul 15 08:00:25.347374 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.347355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmdh8" Jul 15 08:00:25.348946 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.348923 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda203fc45_5d0c_4c8c_9e32_9f6ff42960d0.slice/crio-900d1f0abf7d67eca9cc7a47a5105897e1c353a5e23b83ac9a50423ad84cd4b0 WatchSource:0}: Error finding container 900d1f0abf7d67eca9cc7a47a5105897e1c353a5e23b83ac9a50423ad84cd4b0: Status 404 returned error can't find the container with id 900d1f0abf7d67eca9cc7a47a5105897e1c353a5e23b83ac9a50423ad84cd4b0 Jul 15 08:00:25.355704 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.355687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgl8n" Jul 15 08:00:25.356139 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.356117 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4cff77_0eaf_4210_9839_c8a3cbe90021.slice/crio-9fb726d58dd6003bc859d916164dee1920c5bdfcf4764a895d71ccbac7c7c3bc WatchSource:0}: Error finding container 9fb726d58dd6003bc859d916164dee1920c5bdfcf4764a895d71ccbac7c7c3bc: Status 404 returned error can't find the container with id 9fb726d58dd6003bc859d916164dee1920c5bdfcf4764a895d71ccbac7c7c3bc Jul 15 08:00:25.362653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.362624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sxggh" Jul 15 08:00:25.365179 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.365072 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf327ca_0c00_409e_9a4b_d808203fcdfd.slice/crio-ff226eb9b67502d245be760bda3c1d9bed6e29b22d3459eb068bd8501db5ba25 WatchSource:0}: Error finding container ff226eb9b67502d245be760bda3c1d9bed6e29b22d3459eb068bd8501db5ba25: Status 404 returned error can't find the container with id ff226eb9b67502d245be760bda3c1d9bed6e29b22d3459eb068bd8501db5ba25 Jul 15 08:00:25.367657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.367641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:25.371488 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.371308 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 08:00:25.371665 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.371648 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa330738_6386_40f9_8d31_b823ef2739cc.slice/crio-4e78fda7d438fd27b8c70d47d8d130b5a3238f2e6e92f01d5f6cfdefdc5ff05b WatchSource:0}: Error finding container 4e78fda7d438fd27b8c70d47d8d130b5a3238f2e6e92f01d5f6cfdefdc5ff05b: Status 404 returned error can't find the container with id 4e78fda7d438fd27b8c70d47d8d130b5a3238f2e6e92f01d5f6cfdefdc5ff05b Jul 15 08:00:25.377326 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:25.377297 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5fddbb_ff3c_4a37_9dbd_f57ff6b26ad9.slice/crio-d3c5284cbe1f6ffe3fb51fd67e3df22a865b111c88fad9f4173d58f43fb83ee9 WatchSource:0}: Error finding container d3c5284cbe1f6ffe3fb51fd67e3df22a865b111c88fad9f4173d58f43fb83ee9: Status 404 returned error can't find the container with id d3c5284cbe1f6ffe3fb51fd67e3df22a865b111c88fad9f4173d58f43fb83ee9 Jul 15 08:00:25.623657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.623577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:25.623657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.623635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:25.623848 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.623731 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:25.623848 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.623742 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:25.623848 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.623799 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:26.623783635 +0000 UTC m=+4.043528413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:25.623848 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.623816 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:26.62380752 +0000 UTC m=+4.043552298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:25.724921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:25.724890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:25.725126 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.725040 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:25.725126 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.725056 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:25.725126 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.725067 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:25.725126 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:25.725119 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:26.725103503 +0000 UTC m=+4.144848299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:26.052513 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.052472 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 07:55:24 +0000 UTC" deadline="2028-03-21 21:13:49.662298314 +0000 UTC" Jul 15 08:00:26.053130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.052512 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14773h13m23.609790436s" Jul 15 08:00:26.117738 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.117699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sxggh" event={"ID":"aa330738-6386-40f9-8d31-b823ef2739cc","Type":"ContainerStarted","Data":"4e78fda7d438fd27b8c70d47d8d130b5a3238f2e6e92f01d5f6cfdefdc5ff05b"} Jul 15 08:00:26.119967 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.119940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmdh8" event={"ID":"1d4cff77-0eaf-4210-9839-c8a3cbe90021","Type":"ContainerStarted","Data":"9fb726d58dd6003bc859d916164dee1920c5bdfcf4764a895d71ccbac7c7c3bc"} Jul 15 08:00:26.121854 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.121830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" event={"ID":"75cda433-8419-4e55-abd6-ebded8a827bf","Type":"ContainerStarted","Data":"cca4ce325ea2bc3994de31968642d9f814810c8be3e143a5b894b05144ce9537"} Jul 15 08:00:26.123415 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.123373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhb8r" event={"ID":"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9","Type":"ContainerStarted","Data":"d3c5284cbe1f6ffe3fb51fd67e3df22a865b111c88fad9f4173d58f43fb83ee9"} Jul 15 08:00:26.124967 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.124942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgl8n" event={"ID":"eaf327ca-0c00-409e-9a4b-d808203fcdfd","Type":"ContainerStarted","Data":"ff226eb9b67502d245be760bda3c1d9bed6e29b22d3459eb068bd8501db5ba25"} Jul 15 08:00:26.128109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.128064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerStarted","Data":"900d1f0abf7d67eca9cc7a47a5105897e1c353a5e23b83ac9a50423ad84cd4b0"} Jul 15 08:00:26.129706 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.129677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"35e1b0acbe7be1bbec0f01ae5a1e681f175ae0a9981403b18d57ee1fc868f359"} Jul 15 08:00:26.131165 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.131127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jhvd" event={"ID":"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6","Type":"ContainerStarted","Data":"c9097537e0c7e74d01a127f96086ff29371d49fc599023956e62989f79faf04a"} Jul 15 08:00:26.133066 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.133024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-22kqk" event={"ID":"d65906e9-c2f0-408c-9802-45585194971c","Type":"ContainerStarted","Data":"02df9b6d2125b41bd14df7934e2ae0d59b25ea201a80257e34ac4047e035e931"} Jul 15 08:00:26.631427 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.631382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:26.631586 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.631468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:26.631586 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.631577 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:26.631699 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.631632 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:28.631614226 +0000 UTC m=+6.051359008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:26.632007 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.631992 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:26.632122 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.632044 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:28.632028827 +0000 UTC m=+6.051773606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:26.732837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:26.732802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:26.733035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.732972 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:26.733035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.732988 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:26.733035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.732999 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:26.733180 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:26.733051 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:28.733034948 +0000 UTC m=+6.152779730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:27.053039 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:27.052981 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 07:55:24 +0000 UTC" deadline="2028-04-26 06:48:11.216742156 +0000 UTC" Jul 15 08:00:27.053039 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:27.053016 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15622h47m44.163730229s" Jul 15 08:00:27.120369 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:27.120338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:27.120571 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:27.120545 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:27.121184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:27.121167 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:27.121447 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:27.121428 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:27.121549 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:27.121531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:27.121675 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:27.121655 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:28.651754 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:28.651715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:28.652222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:28.651793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:28.652222 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.651890 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:28.652222 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.651918 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:28.652222 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.651945 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:32.651930803 +0000 UTC m=+10.071675584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:28.652222 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.651966 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:32.651951051 +0000 UTC m=+10.071695835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:28.752908 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:28.752728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:28.752908 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.752912 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:28.753148 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.752934 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:28.753148 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.752946 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:28.753148 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:28.753003 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:32.752986604 +0000 UTC m=+10.172731398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:29.111317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:29.111284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:29.111502 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:29.111284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:29.111502 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:29.111436 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:29.111502 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:29.111448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:29.111670 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:29.111538 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:29.111670 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:29.111625 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:31.111460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:31.111428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:31.111460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:31.111453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:31.112035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:31.111428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:31.112035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:31.111568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:31.112035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:31.111639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:31.112035 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:31.111738 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:32.688732 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:32.688683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:32.689243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:32.688766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:32.689243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.688848 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:32.689243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.688885 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:32.689243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.688928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:40.688907704 +0000 UTC m=+18.108652490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:32.689243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.688951 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:40.688940546 +0000 UTC m=+18.108685328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:32.789811 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:32.789769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:32.790007 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.789989 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:32.790104 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.790013 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:32.790104 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.790026 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:32.790104 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:32.790090 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:40.790073357 +0000 UTC m=+18.209818152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:33.111896 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:33.111858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:33.112069 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:33.111979 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:33.112128 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:33.112076 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:33.112210 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:33.112163 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:33.112327 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:33.112274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:33.112384 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:33.112352 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:35.110829 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:35.110797 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:35.111280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:35.110797 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:35.111280 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:35.110929 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:35.111280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:35.110979 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:35.111280 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:35.111068 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:35.111280 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:35.111158 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:37.110806 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:37.110771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:37.111265 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:37.110782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:37.111265 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:37.110896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:37.111265 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:37.110891 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:37.111265 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:37.110998 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:37.111265 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:37.111105 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:39.110862 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:39.110828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:39.110862 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:39.110842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:39.111299 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:39.110831 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:39.111299 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:39.110962 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:39.111299 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:39.111011 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:39.111299 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:39.111095 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:40.748354 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:40.748153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:40.748909 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:40.748386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:40.748909 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.748321 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:40.748909 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.748494 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:40.748909 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.748510 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:00:56.748492995 +0000 UTC m=+34.168237776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:40.748909 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.748537 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:56.74852081 +0000 UTC m=+34.168265592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:40.849267 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:40.849232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:40.849441 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.849415 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:40.849501 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.849441 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:40.849501 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.849456 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:40.849578 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:40.849513 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:56.849497201 +0000 UTC m=+34.269241986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:41.111307 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:41.111213 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:41.111307 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:41.111245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:41.111537 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:41.111350 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:41.111537 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:41.111421 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:41.111537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:41.111478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:41.111667 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:41.111552 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:43.111681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:43.111644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:43.112138 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:43.111766 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:43.112138 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:43.111788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:43.112138 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:43.111812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:43.112138 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:43.111885 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:43.112138 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:43.111988 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:44.184393 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.184353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgl8n" event={"ID":"eaf327ca-0c00-409e-9a4b-d808203fcdfd","Type":"ContainerStarted","Data":"daa618973517000100b2bb326629d290643dd3787735104b00a5b3111e51d052"} Jul 15 08:00:44.188962 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.188896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-22kqk" event={"ID":"d65906e9-c2f0-408c-9802-45585194971c","Type":"ContainerStarted","Data":"f1a257befa0e34b5f54953f3d3b795b62b1b44a1235c1116681bff7099581426"} Jul 15 08:00:44.193657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.193613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"b0a59e8f4957b010cc06fb503bb4835a","Type":"ContainerStarted","Data":"e1233f1eab0eff453da6643514e95fe88329a3308b0a487c23dd4663ca3ca7a3"} Jul 15 08:00:44.205586 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.205525 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kgl8n" podStartSLOduration=2.949067987 podStartE2EDuration="21.205503205s" totalImagesPullingTime="18.256435218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:44.204740071 +0000 UTC m=+21.624484900" watchObservedRunningTime="2026-07-15 08:00:44.205503205 +0000 UTC m=+21.625248009" Jul 15 08:00:44.220453 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.219813 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" podStartSLOduration=20.21979202 podStartE2EDuration="20.21979202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:44.219348212 +0000 UTC m=+21.639093016" watchObservedRunningTime="2026-07-15 08:00:44.21979202 +0000 UTC m=+21.639536824" Jul 15 08:00:44.239099 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:44.239054 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-22kqk" podStartSLOduration=3.96912012 podStartE2EDuration="22.23903903s" totalImagesPullingTime="18.26991891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:44.23849405 +0000 UTC m=+21.658238871" watchObservedRunningTime="2026-07-15 08:00:44.23903903 +0000 UTC m=+21.658783833" Jul 15 08:00:45.110951 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.110648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:45.111279 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.110674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:45.111279 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:45.111019 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:45.111279 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:45.111085 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:45.111279 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.110739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:45.111279 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:45.111206 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:45.197160 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.197122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sxggh" event={"ID":"aa330738-6386-40f9-8d31-b823ef2739cc","Type":"ContainerStarted","Data":"74cc15fe308f4063ffdedfaf44185702115d919b058c20951b2e3410dd0f13f4"} Jul 15 08:00:45.198534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.198508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmdh8" event={"ID":"1d4cff77-0eaf-4210-9839-c8a3cbe90021","Type":"ContainerStarted","Data":"8d7ab43638d0b0b8489ba39d46e1e4e7e0817a91a5d09c23216048171ef0004c"} Jul 15 08:00:45.199766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.199747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" event={"ID":"75cda433-8419-4e55-abd6-ebded8a827bf","Type":"ContainerStarted","Data":"027262361c414c0a6c9ba657c3139711fe28ad298be0883d9ca9b3eded00a17a"} Jul 15 08:00:45.201143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.201118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhb8r" event={"ID":"2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9","Type":"ContainerStarted","Data":"4bc5f667d0474f66ae0af415ca3bb467f0633af9666a4c012a5d670e006101b4"} Jul 15 08:00:45.202566 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.202541 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="05275056b52bfddd6e60f91febb9756961622d2268fa78bfe74ea3e4a65a8a63" exitCode=0 Jul 15 08:00:45.202673 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.202625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"05275056b52bfddd6e60f91febb9756961622d2268fa78bfe74ea3e4a65a8a63"} Jul 15 08:00:45.205387 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205363 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:00:45.205737 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205718 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4f8670d-87f7-4d3d-af8a-10a9afd44afe" containerID="db2b09d26889501e1c387b884530448210a3ffc85b931ccf93de8ec101d6ea77" exitCode=1 Jul 15 08:00:45.205822 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"0201114b11d27e122e8f17bd5332fb63bac1c27c707ac7c8ae4b793e49ca1403"} Jul 15 08:00:45.205863 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"a8af8f87f5c26afd6d3e65f62d352f3bd979727020ffb790efed2ad413ef581b"} Jul 15 08:00:45.205904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"16516cac01a98d135eae6576254f7fd7b0e507cc3fddce08ec6999bc314eafbe"} Jul 15 08:00:45.205946 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"3cbce5ac5f8f53498febe1098cbea3b6ad2e0feeeeaa3ec940252d117d2cb0ea"} Jul 15 08:00:45.205995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.205983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerDied","Data":"db2b09d26889501e1c387b884530448210a3ffc85b931ccf93de8ec101d6ea77"} Jul 15 08:00:45.206029 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.206017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"b6334145cf76ca8a0c0fa11543f6c3cc6c187e5725b268150bad6c5bf1add792"} Jul 15 08:00:45.207016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.206998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jhvd" event={"ID":"a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6","Type":"ContainerStarted","Data":"0486150d669b13c1cae277a5a325247d697d6d8787594c907526c31f4aa0a74e"} Jul 15 08:00:45.208515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.208492 2565 generic.go:356] "Generic (PLEG): container finished" podID="181f6c79728673db7ccb0cd897d643da" containerID="d91baa4173badfd0ec6a8214533bd12e06ab187ec92d455b4cb36625b5f8e0f1" exitCode=0 Jul 15 08:00:45.208597 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.208545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"181f6c79728673db7ccb0cd897d643da","Type":"ContainerDied","Data":"d91baa4173badfd0ec6a8214533bd12e06ab187ec92d455b4cb36625b5f8e0f1"} Jul 15 08:00:45.218678 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.218644 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sxggh" podStartSLOduration=4.23341386 podStartE2EDuration="22.218632881s" totalImagesPullingTime="17.985219021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:45.217610958 +0000 UTC m=+22.637355780" watchObservedRunningTime="2026-07-15 08:00:45.218632881 +0000 UTC m=+22.638377683" Jul 15 08:00:45.249546 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.249509 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9jhvd" podStartSLOduration=5.212288756 podStartE2EDuration="23.24949604s" totalImagesPullingTime="18.037207284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:45.249378044 +0000 UTC m=+22.669122848" watchObservedRunningTime="2026-07-15 08:00:45.24949604 +0000 UTC m=+22.669240843" Jul 15 08:00:45.286301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.286258 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhb8r" podStartSLOduration=4.079597442 podStartE2EDuration="22.286236627s" totalImagesPullingTime="18.206639185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:45.285645733 +0000 UTC m=+22.705390527" watchObservedRunningTime="2026-07-15 08:00:45.286236627 +0000 UTC m=+22.705981431" Jul 15 08:00:45.302345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:45.302303 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hmdh8" podStartSLOduration=12.159684917 podStartE2EDuration="22.302284678s" totalImagesPullingTime="10.142599761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:45.302221872 +0000 UTC m=+22.721966677" watchObservedRunningTime="2026-07-15 08:00:45.302284678 +0000 UTC m=+22.722029482" Jul 15 08:00:46.212127 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:46.212083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"181f6c79728673db7ccb0cd897d643da","Type":"ContainerStarted","Data":"e1c049298ae42168351d977978fa4048edcb538b39af2a74de286054b0de018c"} Jul 15 08:00:46.227871 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:46.227815 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" podStartSLOduration=22.227797711 podStartE2EDuration="22.227797711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:46.227514637 +0000 UTC m=+23.647259439" watchObservedRunningTime="2026-07-15 08:00:46.227797711 +0000 UTC m=+23.647542517" Jul 15 08:00:46.683733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:46.683711 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 08:00:46.848173 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:46.848089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:46.848867 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:46.848834 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:47.085331 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.085005 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T08:00:46.683728559Z","UUID":"3cc204c3-e16a-41a7-aadc-c7d9728d6875","Handler":null,"Name":"","Endpoint":""} Jul 15 08:00:47.086552 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.086531 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 08:00:47.086674 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.086571 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 08:00:47.110454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.110374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:47.110583 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.110514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:47.110646 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:47.110512 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:47.110646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.110634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:47.110735 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:47.110633 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:47.110779 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:47.110734 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:47.216466 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.216431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" event={"ID":"75cda433-8419-4e55-abd6-ebded8a827bf","Type":"ContainerStarted","Data":"323c9bafac0749194bf59b49d42c2df4f7bb9ff0df90be0935572462b2eb343a"} Jul 15 08:00:47.219819 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.219799 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:00:47.220254 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.220222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"c7dab308e5d74daf57b83a55d88a5a9e82e2629c02a3af960a145754ebcc6e07"} Jul 15 08:00:47.220340 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.220318 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:47.220889 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:47.220873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhb8r" Jul 15 08:00:49.110769 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:49.110659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:49.110769 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:49.110718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:49.111243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:49.110816 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:49.111243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:49.110833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:49.111243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:49.110935 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:49.111243 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:49.111015 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:50.232763 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.232598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:00:50.233349 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.233131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"0e8cba55afa2dda904c72ef2550306c05036c9cf08d98c81969c10253ff156a4"} Jul 15 08:00:50.233432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.233373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:50.233617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.233602 2565 scope.go:117] "RemoveContainer" containerID="db2b09d26889501e1c387b884530448210a3ffc85b931ccf93de8ec101d6ea77" Jul 15 08:00:50.235252 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.235228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" event={"ID":"75cda433-8419-4e55-abd6-ebded8a827bf","Type":"ContainerStarted","Data":"59793e7d9ce91d02b2f8a4f33ffa3b66b825843920fd038e1b60895d11a91596"} Jul 15 08:00:50.236989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.236963 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="680a5eee32454676b7e4fffa2253699acc8e0f2579628ea7af4583ce7c192b78" exitCode=0 Jul 15 08:00:50.237093 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.237001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"680a5eee32454676b7e4fffa2253699acc8e0f2579628ea7af4583ce7c192b78"} Jul 15 08:00:50.249740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.249721 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:50.278881 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:50.278841 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cgdj" podStartSLOduration=11.936833098 podStartE2EDuration="27.278826268s" totalImagesPullingTime="15.34199317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:50.278577144 +0000 UTC m=+27.698321958" watchObservedRunningTime="2026-07-15 08:00:50.278826268 +0000 UTC m=+27.698571070" Jul 15 08:00:51.110346 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.110291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:51.110518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.110291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:51.110518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.110291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:51.110518 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.110444 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:51.110518 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.110494 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:51.110698 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.110615 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:51.244028 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.244004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:00:51.245300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.244983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" event={"ID":"b4f8670d-87f7-4d3d-af8a-10a9afd44afe","Type":"ContainerStarted","Data":"5430de6b33725d9ba23c476ba6bdecc897f4b7cd9db8d3273dfbc6525d85a368"} Jul 15 08:00:51.245698 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.245679 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:51.245886 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.245861 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:51.262717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.262690 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:00:51.276738 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.276693 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" podStartSLOduration=9.497983575 podStartE2EDuration="28.276679668s" totalImagesPullingTime="18.778696093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:00:51.276368504 +0000 UTC m=+28.696113306" watchObservedRunningTime="2026-07-15 08:00:51.276679668 +0000 UTC m=+28.696424470" Jul 15 08:00:51.925965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.925930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2s78"] Jul 15 08:00:51.926103 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.926029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:51.926150 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.926107 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:51.928420 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.928375 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jh48p"] Jul 15 08:00:51.928543 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.928475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:51.928596 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.928580 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:51.928956 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.928936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ptzbf"] Jul 15 08:00:51.929030 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:51.929020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:51.929127 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:51.929108 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:52.248647 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:52.248613 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="1eedadfdd1fb81f9e8fbf5d28e8c1f968c031c128daef37ccd8df2440864622f" exitCode=0 Jul 15 08:00:52.249003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:52.248704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"1eedadfdd1fb81f9e8fbf5d28e8c1f968c031c128daef37ccd8df2440864622f"} Jul 15 08:00:53.111852 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:53.111635 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:53.111982 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:53.111957 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:53.112041 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:53.111732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:53.112109 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:53.112045 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:53.112109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:53.111697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:53.112203 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:53.112124 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:53.253978 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:53.253942 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="bcf8b5e4fb19d0fc9d5c1ae7d7a72073683fe7c737730c7e6a5ee345dc44e5e5" exitCode=0 Jul 15 08:00:53.254353 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:53.254069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"bcf8b5e4fb19d0fc9d5c1ae7d7a72073683fe7c737730c7e6a5ee345dc44e5e5"} Jul 15 08:00:55.110829 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:55.110790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:55.111447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:55.110790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:55.111447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:55.110790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:55.111447 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:55.110941 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh48p" podUID="44438dfa-d58b-48f6-9002-0bbd27cf84c5" Jul 15 08:00:55.111447 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:55.111027 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2s78" podUID="bbb9d64a-d1b4-4feb-a041-4de44d59beaa" Jul 15 08:00:55.111447 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:55.111097 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ptzbf" podUID="616fe1a6-17ce-4a60-995c-b047e24d6338" Jul 15 08:00:56.777266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.777222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:56.777895 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.777291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:56.777895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.777390 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:56.777895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.777416 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:56.777895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.777481 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret podName:bbb9d64a-d1b4-4feb-a041-4de44d59beaa nodeName:}" failed. No retries permitted until 2026-07-15 08:01:28.777460213 +0000 UTC m=+66.197205047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret") pod "global-pull-secret-syncer-f2s78" (UID: "bbb9d64a-d1b4-4feb-a041-4de44d59beaa") : object "kube-system"/"original-pull-secret" not registered Jul 15 08:00:56.777895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.777501 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:28.777491242 +0000 UTC m=+66.197236022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 08:00:56.782063 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.782039 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeReady" Jul 15 08:00:56.782185 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.782157 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 08:00:56.855174 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.855142 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:00:56.869482 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.869457 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.873003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.872983 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 08:00:56.873384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.873366 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 08:00:56.873515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.873498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 08:00:56.873740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.873704 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wb89\"" Jul 15 08:00:56.877928 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.877905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:56.878109 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.878090 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 08:00:56.878151 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.878117 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 08:00:56.878151 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.878130 2565 projected.go:196] Error preparing data for projected volume kube-api-access-csrxm for pod openshift-network-diagnostics/network-check-target-ptzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:56.878211 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:56.878192 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm podName:616fe1a6-17ce-4a60-995c-b047e24d6338 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:28.878173094 +0000 UTC m=+66.297917889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-csrxm" (UniqueName: "kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm") pod "network-check-target-ptzbf" (UID: "616fe1a6-17ce-4a60-995c-b047e24d6338") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 08:00:56.889683 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.889660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:00:56.900141 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.898740 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 08:00:56.923126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.923097 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-896pl"] Jul 15 08:00:56.944942 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.944922 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8s45p"] Jul 15 08:00:56.945071 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.945056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-896pl" Jul 15 08:00:56.950332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.949833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 08:00:56.950332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.950172 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8tgdm\"" Jul 15 08:00:56.953681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.953550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 08:00:56.955271 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.955050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-896pl"] Jul 15 08:00:56.955271 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.955092 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8s45p"] Jul 15 08:00:56.955271 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.955174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:56.958856 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.958840 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 08:00:56.959066 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.959048 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 08:00:56.959168 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.959152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lkxcp\"" Jul 15 08:00:56.959555 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.959534 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 08:00:56.978689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978810 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978810 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxs76\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978810 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978964 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978964 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978964 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:56.978964 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:56.978931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.079600 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxs76\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.079793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qxj\" (UniqueName: \"kubernetes.io/projected/6a0b8373-62f9-4798-a412-b5ba19e87b42-kube-api-access-66qxj\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.079793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.079793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.079793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.079793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0b8373-62f9-4798-a412-b5ba19e87b42-cert\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f885a2-8fd2-4059-a3a5-988292116572-config-volume\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f885a2-8fd2-4059-a3a5-988292116572-tmp-dir\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.080001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.079991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clt9\" (UniqueName: \"kubernetes.io/projected/55f885a2-8fd2-4059-a3a5-988292116572-kube-api-access-9clt9\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.080273 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.080039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080371 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.080335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080371 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.080359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.080536 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.080426 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:00:57.080536 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.080447 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:00:57.080536 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.080518 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:57.580499651 +0000 UTC m=+35.000244436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:00:57.080966 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.080937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.084550 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.084529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.084643 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.084562 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.095003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.094968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.095878 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.095851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxs76\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.110645 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.110621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:00:57.110816 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.110792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:00:57.110917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.110855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:00:57.113524 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.113505 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 08:00:57.114898 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.114878 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 08:00:57.115056 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.115043 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df2lv\"" Jul 15 08:00:57.115209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.115194 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 08:00:57.115711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.115688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 08:00:57.115861 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.115846 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5x8qz\"" Jul 15 08:00:57.180910 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.180879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qxj\" (UniqueName: \"kubernetes.io/projected/6a0b8373-62f9-4798-a412-b5ba19e87b42-kube-api-access-66qxj\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.181064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.180946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0b8373-62f9-4798-a412-b5ba19e87b42-cert\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.181064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.180972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.181064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.181022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f885a2-8fd2-4059-a3a5-988292116572-config-volume\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.181064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.181059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f885a2-8fd2-4059-a3a5-988292116572-tmp-dir\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.181245 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.181129 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:00:57.181245 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.181193 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:57.681173239 +0000 UTC m=+35.100918021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:00:57.181245 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.181190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clt9\" (UniqueName: \"kubernetes.io/projected/55f885a2-8fd2-4059-a3a5-988292116572-kube-api-access-9clt9\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.181430 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.181394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f885a2-8fd2-4059-a3a5-988292116572-tmp-dir\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.181728 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.181702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55f885a2-8fd2-4059-a3a5-988292116572-config-volume\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.183951 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.183933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0b8373-62f9-4798-a412-b5ba19e87b42-cert\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.193481 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.193461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clt9\" (UniqueName: \"kubernetes.io/projected/55f885a2-8fd2-4059-a3a5-988292116572-kube-api-access-9clt9\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.193631 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.193611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qxj\" (UniqueName: \"kubernetes.io/projected/6a0b8373-62f9-4798-a412-b5ba19e87b42-kube-api-access-66qxj\") pod \"ingress-canary-8s45p\" (UID: \"6a0b8373-62f9-4798-a412-b5ba19e87b42\") " pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.224571 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.224537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhb8r_2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9/konnectivity-agent/0.log" limit=1024 Jul 15 08:00:57.265054 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.265023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8s45p" Jul 15 08:00:57.320886 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.320852 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhb8r_2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9/konnectivity-agent/0.log" limit=1024 Jul 15 08:00:57.585845 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.585807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:57.586119 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.586006 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:00:57.586119 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.586025 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:00:57.586119 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.586084 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:58.586065556 +0000 UTC m=+36.005810338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:00:57.687218 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:57.687183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:57.687397 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.687310 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:00:57.687397 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:57.687369 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:00:58.687352676 +0000 UTC m=+36.107097477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:00:58.551117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:58.550957 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8s45p"] Jul 15 08:00:58.593981 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:58.593955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:00:58.594091 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:58.594066 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:00:58.594091 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:58.594077 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:00:58.594155 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:58.594122 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:00.594110244 +0000 UTC m=+38.013855025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:00:58.611946 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:00:58.611921 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0b8373_62f9_4798_a412_b5ba19e87b42.slice/crio-64c41e1bb2f5cc65299d954a5362c71937463e33e7a82c3c6bf45c1bbe9d3dde WatchSource:0}: Error finding container 64c41e1bb2f5cc65299d954a5362c71937463e33e7a82c3c6bf45c1bbe9d3dde: Status 404 returned error can't find the container with id 64c41e1bb2f5cc65299d954a5362c71937463e33e7a82c3c6bf45c1bbe9d3dde Jul 15 08:00:58.694855 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:58.694833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:00:58.694996 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:58.694977 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:00:58.695044 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:00:58.695039 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:00.695023912 +0000 UTC m=+38.114768697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:00:59.268921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:59.268884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8s45p" event={"ID":"6a0b8373-62f9-4798-a412-b5ba19e87b42","Type":"ContainerStarted","Data":"64c41e1bb2f5cc65299d954a5362c71937463e33e7a82c3c6bf45c1bbe9d3dde"} Jul 15 08:00:59.271451 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:59.271425 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="256f6e32cb995ec03a597d915c1a0786b62743e638ca2771757f2865fd83d066" exitCode=0 Jul 15 08:00:59.271581 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:00:59.271456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"256f6e32cb995ec03a597d915c1a0786b62743e638ca2771757f2865fd83d066"} Jul 15 08:01:00.277870 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.277835 2565 generic.go:356] "Generic (PLEG): container finished" podID="a203fc45-5d0c-4c8c-9e32-9f6ff42960d0" containerID="baeae4e7988557fef392e8571c3397c55f91bbf60707eeb37b2a819ff642e56e" exitCode=0 Jul 15 08:01:00.278352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.277900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerDied","Data":"baeae4e7988557fef392e8571c3397c55f91bbf60707eeb37b2a819ff642e56e"} Jul 15 08:01:00.468344 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.468091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhb8r_2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:00.557295 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.557212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhb8r_2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9/konnectivity-agent/0.log" limit=1024 Jul 15 08:01:00.611790 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.611754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:00.611966 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:00.611898 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:01:00.611966 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:00.611918 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:01:00.612079 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:00.611984 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:04.611965268 +0000 UTC m=+42.031710055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:01:00.712503 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:00.712461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:01:00.712673 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:00.712644 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:00.712737 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:00.712725 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:04.712706058 +0000 UTC m=+42.132450859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:01:01.284580 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:01.284541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" event={"ID":"a203fc45-5d0c-4c8c-9e32-9f6ff42960d0","Type":"ContainerStarted","Data":"b49173f1c077e94664688a31be448793dc2a1f9e0fdd02b180edc4dd801cb56e"} Jul 15 08:01:01.311548 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:01.311484 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sw7h4" podStartSLOduration=8.086181249 podStartE2EDuration="38.311464049s" totalImagesPullingTime="30.2252828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:01.310457511 +0000 UTC m=+38.730202332" watchObservedRunningTime="2026-07-15 08:01:01.311464049 +0000 UTC m=+38.731208851" Jul 15 08:01:02.288465 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:02.288421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8s45p" event={"ID":"6a0b8373-62f9-4798-a412-b5ba19e87b42","Type":"ContainerStarted","Data":"890d40826f09303c28fe8d25cd60e71eb332003209ed4a151d8ffc5438399d79"} Jul 15 08:01:02.303952 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:02.303893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8s45p" podStartSLOduration=2.804282846 podStartE2EDuration="6.30387353s" totalImagesPullingTime="3.499590684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:02.303631128 +0000 UTC m=+39.723375932" watchObservedRunningTime="2026-07-15 08:01:02.30387353 +0000 UTC m=+39.723618330" Jul 15 08:01:04.636939 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:04.636899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:04.637298 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:04.637042 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:01:04.637298 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:04.637061 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:01:04.637298 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:04.637115 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:12.637100779 +0000 UTC m=+50.056845559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:01:04.737493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:04.737453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:01:04.737648 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:04.737600 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:04.737686 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:04.737663 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:12.737644661 +0000 UTC m=+50.157389444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:01:09.862543 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.862508 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng"] Jul 15 08:01:09.899798 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.899771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng"] Jul 15 08:01:09.899929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.899856 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:09.907562 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.907544 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 15 08:01:09.907672 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.907655 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 15 08:01:09.908370 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.908355 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-t6f2p\"" Jul 15 08:01:09.908496 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.908479 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:09.908834 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.908816 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 15 08:01:09.971232 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.971205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d7f2ca-b85d-4084-9138-4d71c041be8a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:09.971337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.971255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngb8q\" (UniqueName: \"kubernetes.io/projected/f8d7f2ca-b85d-4084-9138-4d71c041be8a-kube-api-access-ngb8q\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:09.971337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:09.971307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d7f2ca-b85d-4084-9138-4d71c041be8a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.003066 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.003045 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs"] Jul 15 08:01:10.026165 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.026143 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f8844f56-rsmbf"] Jul 15 08:01:10.026287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.026269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.028842 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.028782 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 15 08:01:10.029727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.029708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.030339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.030316 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.031004 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.030987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2hc7p\"" Jul 15 08:01:10.043479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.043461 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6"] Jul 15 08:01:10.043592 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.043578 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.047305 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.047285 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cwj6j\"" Jul 15 08:01:10.047700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.047644 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 15 08:01:10.048805 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.048784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 15 08:01:10.048925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.048808 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.048925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.048808 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.048925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.048810 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 15 08:01:10.049082 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.048796 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 15 08:01:10.065541 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.065523 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sf7fl"] Jul 15 08:01:10.065645 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.065631 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" Jul 15 08:01:10.069134 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.069115 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.069230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.069117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.069382 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.069367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hg6d7\"" Jul 15 08:01:10.071753 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6n76\" (UniqueName: \"kubernetes.io/projected/7813420f-c6bb-4140-b981-6a53eae98b4c-kube-api-access-q6n76\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.071866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb8l\" (UniqueName: \"kubernetes.io/projected/7f3d67b1-ae4c-4494-b8be-54edcc5977c9-kube-api-access-4qb8l\") pod \"volume-data-source-validator-78495597f6-5kfn6\" (UID: \"7f3d67b1-ae4c-4494-b8be-54edcc5977c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" Jul 15 08:01:10.071866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.071866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.071866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw9g\" (UniqueName: \"kubernetes.io/projected/55aeea10-575f-4ca6-b899-02f564240e75-kube-api-access-tdw9g\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.072077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d7f2ca-b85d-4084-9138-4d71c041be8a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.072077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.071900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-default-certificate\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.072184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.072086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngb8q\" (UniqueName: \"kubernetes.io/projected/f8d7f2ca-b85d-4084-9138-4d71c041be8a-kube-api-access-ngb8q\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.072184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.072138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d7f2ca-b85d-4084-9138-4d71c041be8a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.072184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.072157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.072184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.072172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-stats-auth\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.072380 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.072353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d7f2ca-b85d-4084-9138-4d71c041be8a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.076025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.075998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d7f2ca-b85d-4084-9138-4d71c041be8a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.093876 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.093857 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l"] Jul 15 08:01:10.093982 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.093967 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.100435 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.100394 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.100435 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.100433 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.100574 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.100451 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 15 08:01:10.100574 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.100474 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-djmwt\"" Jul 15 08:01:10.103020 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.103007 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 15 08:01:10.109075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.109055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngb8q\" (UniqueName: \"kubernetes.io/projected/f8d7f2ca-b85d-4084-9138-4d71c041be8a-kube-api-access-ngb8q\") pod \"kube-storage-version-migrator-operator-799cb7f958-4gsng\" (UID: \"f8d7f2ca-b85d-4084-9138-4d71c041be8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.117218 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 15 08:01:10.117386 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs"] Jul 15 08:01:10.117483 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117397 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8844f56-rsmbf"] Jul 15 08:01:10.117483 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.117590 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6"] Jul 15 08:01:10.117590 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117520 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sf7fl"] Jul 15 08:01:10.117590 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.117536 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l"] Jul 15 08:01:10.118300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.118281 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w"] Jul 15 08:01:10.120656 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.120636 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 15 08:01:10.120656 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.120650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.120781 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.120710 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.121136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.121122 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vbw7m\"" Jul 15 08:01:10.121180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.121126 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 15 08:01:10.137586 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.137569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.139153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.139137 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w"] Jul 15 08:01:10.142590 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.142572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 15 08:01:10.142732 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.142610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.143091 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.143071 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hgl68\"" Jul 15 08:01:10.143866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.143787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 15 08:01:10.143866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.143794 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.173130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb48dc08-fe83-496b-96ad-9889a57b224f-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.173225 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkltr\" (UniqueName: \"kubernetes.io/projected/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-kube-api-access-lkltr\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.173225 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-serving-cert\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.173321 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n76\" (UniqueName: \"kubernetes.io/projected/7813420f-c6bb-4140-b981-6a53eae98b4c-kube-api-access-q6n76\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.173373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.173373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb8l\" (UniqueName: \"kubernetes.io/projected/7f3d67b1-ae4c-4494-b8be-54edcc5977c9-kube-api-access-4qb8l\") pod \"volume-data-source-validator-78495597f6-5kfn6\" (UID: \"7f3d67b1-ae4c-4494-b8be-54edcc5977c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" Jul 15 08:01:10.173499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.173499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-config\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.173499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-snapshots\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xs6\" (UniqueName: \"kubernetes.io/projected/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-kube-api-access-k5xs6\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.173617 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:10.67360341 +0000 UTC m=+48.093348195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw9g\" (UniqueName: \"kubernetes.io/projected/55aeea10-575f-4ca6-b899-02f564240e75-kube-api-access-tdw9g\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.173653 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.173650 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-tmp\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.173687 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:10.673672359 +0000 UTC m=+48.093417141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-default-certificate\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwzv\" (UniqueName: \"kubernetes.io/projected/fb48dc08-fe83-496b-96ad-9889a57b224f-kube-api-access-rqwzv\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.173869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.174054 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.174054 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.173910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-stats-auth\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.174054 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.173946 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 08:01:10.174054 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.173987 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls podName:7813420f-c6bb-4140-b981-6a53eae98b4c nodeName:}" failed. No retries permitted until 2026-07-15 08:01:10.673972754 +0000 UTC m=+48.093717535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls") pod "cluster-samples-operator-65db858894-5qffs" (UID: "7813420f-c6bb-4140-b981-6a53eae98b4c") : secret "samples-operator-tls" not found Jul 15 08:01:10.175988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.175970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-stats-auth\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.176065 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.176016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-default-certificate\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.202454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.202431 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-r542h"] Jul 15 08:01:10.203754 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.203732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb8l\" (UniqueName: \"kubernetes.io/projected/7f3d67b1-ae4c-4494-b8be-54edcc5977c9-kube-api-access-4qb8l\") pod \"volume-data-source-validator-78495597f6-5kfn6\" (UID: \"7f3d67b1-ae4c-4494-b8be-54edcc5977c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" Jul 15 08:01:10.203832 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.203784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6n76\" (UniqueName: \"kubernetes.io/projected/7813420f-c6bb-4140-b981-6a53eae98b4c-kube-api-access-q6n76\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.208078 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.208061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" Jul 15 08:01:10.210264 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.210241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw9g\" (UniqueName: \"kubernetes.io/projected/55aeea10-575f-4ca6-b899-02f564240e75-kube-api-access-tdw9g\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.223213 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.223195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-r542h"] Jul 15 08:01:10.223280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.223262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.225368 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.225352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 15 08:01:10.226284 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.226266 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nk5lj\"" Jul 15 08:01:10.229088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.229066 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:01:10.229186 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.229152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 15 08:01:10.230774 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.230759 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 15 08:01:10.238011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.237984 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 15 08:01:10.274848 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.274819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xs6\" (UniqueName: \"kubernetes.io/projected/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-kube-api-access-k5xs6\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.274860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-trusted-ca\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.274910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7hj\" (UniqueName: \"kubernetes.io/projected/34bf1e10-21d6-4ea5-8abe-30009956d7ce-kube-api-access-6t7hj\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.274933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-tmp\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.274970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwzv\" (UniqueName: \"kubernetes.io/projected/fb48dc08-fe83-496b-96ad-9889a57b224f-kube-api-access-rqwzv\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb48dc08-fe83-496b-96ad-9889a57b224f-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkltr\" (UniqueName: \"kubernetes.io/projected/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-kube-api-access-lkltr\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-serving-cert\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.275204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34bf1e10-21d6-4ea5-8abe-30009956d7ce-serving-cert\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.276558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-config\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.276558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-config\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.276558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-snapshots\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.276558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.275371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.276558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.276524 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.277358 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.276861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-tmp\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.277470 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.277415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.277693 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.277672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-config\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.278063 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.277860 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:10.278063 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.277945 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:10.777915416 +0000 UTC m=+48.197660198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:10.278224 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.278059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-snapshots\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.278341 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.278316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb48dc08-fe83-496b-96ad-9889a57b224f-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.279777 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.279758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-serving-cert\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.280060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.280037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.285256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.285226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xs6\" (UniqueName: \"kubernetes.io/projected/a0fe38cf-6ff3-4965-8bdf-2472b4abb40b-kube-api-access-k5xs6\") pod \"insights-operator-68d9dd94cf-sf7fl\" (UID: \"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b\") " pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.291105 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.291071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwzv\" (UniqueName: \"kubernetes.io/projected/fb48dc08-fe83-496b-96ad-9889a57b224f-kube-api-access-rqwzv\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.291596 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.291547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkltr\" (UniqueName: \"kubernetes.io/projected/fba0308b-5cb6-4d38-bc52-f3d2aceb70cb-kube-api-access-lkltr\") pod \"service-ca-operator-7695fc6c8c-mn54l\" (UID: \"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.362572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.362538 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng"] Jul 15 08:01:10.363060 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:10.363037 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d7f2ca_b85d_4084_9138_4d71c041be8a.slice/crio-69a3e79230d733b5e6376abc638e2de6125bbdfe78b2ddc6b3d81589555ec171 WatchSource:0}: Error finding container 69a3e79230d733b5e6376abc638e2de6125bbdfe78b2ddc6b3d81589555ec171: Status 404 returned error can't find the container with id 69a3e79230d733b5e6376abc638e2de6125bbdfe78b2ddc6b3d81589555ec171 Jul 15 08:01:10.373744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.373697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" Jul 15 08:01:10.376028 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-trusted-ca\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.376096 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7hj\" (UniqueName: \"kubernetes.io/projected/34bf1e10-21d6-4ea5-8abe-30009956d7ce-kube-api-access-6t7hj\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.376149 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34bf1e10-21d6-4ea5-8abe-30009956d7ce-serving-cert\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.376149 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-config\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.376741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-config\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.376849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.376827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34bf1e10-21d6-4ea5-8abe-30009956d7ce-trusted-ca\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.378284 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.378269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34bf1e10-21d6-4ea5-8abe-30009956d7ce-serving-cert\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.388471 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.388453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7hj\" (UniqueName: \"kubernetes.io/projected/34bf1e10-21d6-4ea5-8abe-30009956d7ce-kube-api-access-6t7hj\") pod \"console-operator-7866497494-r542h\" (UID: \"34bf1e10-21d6-4ea5-8abe-30009956d7ce\") " pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.402281 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.402259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" Jul 15 08:01:10.425233 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.425212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" Jul 15 08:01:10.532852 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.532817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:10.544840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.544812 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6"] Jul 15 08:01:10.548457 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:10.548391 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3d67b1_ae4c_4494_b8be_54edcc5977c9.slice/crio-540fc02bae6aa137e08666e59fb8a00301dae1d9495bba151e677cc82d8872fd WatchSource:0}: Error finding container 540fc02bae6aa137e08666e59fb8a00301dae1d9495bba151e677cc82d8872fd: Status 404 returned error can't find the container with id 540fc02bae6aa137e08666e59fb8a00301dae1d9495bba151e677cc82d8872fd Jul 15 08:01:10.567903 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.567879 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sf7fl"] Jul 15 08:01:10.569756 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:10.569731 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fe38cf_6ff3_4965_8bdf_2472b4abb40b.slice/crio-37f9e9419189d435123c24d14d2fb5912fbb3d9bdb0bec028ef4c9e5ed2128b4 WatchSource:0}: Error finding container 37f9e9419189d435123c24d14d2fb5912fbb3d9bdb0bec028ef4c9e5ed2128b4: Status 404 returned error can't find the container with id 37f9e9419189d435123c24d14d2fb5912fbb3d9bdb0bec028ef4c9e5ed2128b4 Jul 15 08:01:10.599301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.599273 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l"] Jul 15 08:01:10.602932 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:10.602893 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba0308b_5cb6_4d38_bc52_f3d2aceb70cb.slice/crio-92a186cc0acc41cff26e5b2672c789c1275c40df4969916fd55be12aeebb6e61 WatchSource:0}: Error finding container 92a186cc0acc41cff26e5b2672c789c1275c40df4969916fd55be12aeebb6e61: Status 404 returned error can't find the container with id 92a186cc0acc41cff26e5b2672c789c1275c40df4969916fd55be12aeebb6e61 Jul 15 08:01:10.679132 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.679104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.679245 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.679166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:10.679286 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.679252 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 08:01:10.679317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.679283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:10.679317 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.679307 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls podName:7813420f-c6bb-4140-b981-6a53eae98b4c nodeName:}" failed. No retries permitted until 2026-07-15 08:01:11.67929184 +0000 UTC m=+49.099036621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls") pod "cluster-samples-operator-65db858894-5qffs" (UID: "7813420f-c6bb-4140-b981-6a53eae98b4c") : secret "samples-operator-tls" not found Jul 15 08:01:10.679418 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.679340 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:10.679418 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.679377 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:11.679362459 +0000 UTC m=+49.099107240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:10.679418 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.679395 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:11.679386714 +0000 UTC m=+49.099131500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:10.694250 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.694228 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-r542h"] Jul 15 08:01:10.697761 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:10.697738 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bf1e10_21d6_4ea5_8abe_30009956d7ce.slice/crio-8ea1e8cfc1101a5b722ff0c6b527534083e42ec29342772331ae1bbc55199bc1 WatchSource:0}: Error finding container 8ea1e8cfc1101a5b722ff0c6b527534083e42ec29342772331ae1bbc55199bc1: Status 404 returned error can't find the container with id 8ea1e8cfc1101a5b722ff0c6b527534083e42ec29342772331ae1bbc55199bc1 Jul 15 08:01:10.780180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:10.780157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:10.780310 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.780294 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:10.780354 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:10.780348 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:11.780333901 +0000 UTC m=+49.200078682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:11.310854 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.310822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" event={"ID":"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb","Type":"ContainerStarted","Data":"92a186cc0acc41cff26e5b2672c789c1275c40df4969916fd55be12aeebb6e61"} Jul 15 08:01:11.311963 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.311940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" event={"ID":"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b","Type":"ContainerStarted","Data":"37f9e9419189d435123c24d14d2fb5912fbb3d9bdb0bec028ef4c9e5ed2128b4"} Jul 15 08:01:11.313025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.313004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r542h" event={"ID":"34bf1e10-21d6-4ea5-8abe-30009956d7ce","Type":"ContainerStarted","Data":"8ea1e8cfc1101a5b722ff0c6b527534083e42ec29342772331ae1bbc55199bc1"} Jul 15 08:01:11.314089 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.314066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" event={"ID":"7f3d67b1-ae4c-4494-b8be-54edcc5977c9","Type":"ContainerStarted","Data":"540fc02bae6aa137e08666e59fb8a00301dae1d9495bba151e677cc82d8872fd"} Jul 15 08:01:11.314965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.314948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" event={"ID":"f8d7f2ca-b85d-4084-9138-4d71c041be8a","Type":"ContainerStarted","Data":"69a3e79230d733b5e6376abc638e2de6125bbdfe78b2ddc6b3d81589555ec171"} Jul 15 08:01:11.687534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.687272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:11.687534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.687393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:11.687534 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.687422 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 08:01:11.687534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.687476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:11.687534 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.687499 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls podName:7813420f-c6bb-4140-b981-6a53eae98b4c nodeName:}" failed. No retries permitted until 2026-07-15 08:01:13.687479542 +0000 UTC m=+51.107224327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls") pod "cluster-samples-operator-65db858894-5qffs" (UID: "7813420f-c6bb-4140-b981-6a53eae98b4c") : secret "samples-operator-tls" not found Jul 15 08:01:11.687898 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.687569 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:11.687898 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.687578 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:13.687554011 +0000 UTC m=+51.107298805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:11.687898 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.687625 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:13.687611163 +0000 UTC m=+51.107355947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:11.788657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.788617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:11.788875 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.788844 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:11.788959 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:11.788918 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:13.788897161 +0000 UTC m=+51.208641944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:11.966624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.965489 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl"] Jul 15 08:01:11.987110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.986682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl"] Jul 15 08:01:11.987110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.986790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" Jul 15 08:01:11.991133 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:11.990707 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q2flt\"" Jul 15 08:01:12.092157 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.092116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9778\" (UniqueName: \"kubernetes.io/projected/f54e03d5-92ec-4a3e-a850-d70351483a37-kube-api-access-d9778\") pod \"network-check-source-869dcfcdcd-f2rdl\" (UID: \"f54e03d5-92ec-4a3e-a850-d70351483a37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" Jul 15 08:01:12.194119 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.193722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9778\" (UniqueName: \"kubernetes.io/projected/f54e03d5-92ec-4a3e-a850-d70351483a37-kube-api-access-d9778\") pod \"network-check-source-869dcfcdcd-f2rdl\" (UID: \"f54e03d5-92ec-4a3e-a850-d70351483a37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" Jul 15 08:01:12.210427 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.210371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9778\" (UniqueName: \"kubernetes.io/projected/f54e03d5-92ec-4a3e-a850-d70351483a37-kube-api-access-d9778\") pod \"network-check-source-869dcfcdcd-f2rdl\" (UID: \"f54e03d5-92ec-4a3e-a850-d70351483a37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" Jul 15 08:01:12.301045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.300819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" Jul 15 08:01:12.677178 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.677145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl"] Jul 15 08:01:12.681624 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:12.681587 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54e03d5_92ec_4a3e_a850_d70351483a37.slice/crio-2f9d8b6edc6b5504f2f81ae97f95ee3820ee2778182537bfdb823d397bcca15c WatchSource:0}: Error finding container 2f9d8b6edc6b5504f2f81ae97f95ee3820ee2778182537bfdb823d397bcca15c: Status 404 returned error can't find the container with id 2f9d8b6edc6b5504f2f81ae97f95ee3820ee2778182537bfdb823d397bcca15c Jul 15 08:01:12.699786 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.699724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:12.699895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:12.699847 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 08:01:12.699895 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:12.699868 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66dcc4df8-htzcn: secret "image-registry-tls" not found Jul 15 08:01:12.699991 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:12.699926 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls podName:5dfe29f9-411a-4def-9aaf-41cbea161e06 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:28.699903001 +0000 UTC m=+66.119647787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls") pod "image-registry-66dcc4df8-htzcn" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06") : secret "image-registry-tls" not found Jul 15 08:01:12.800412 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:12.800365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:01:12.800559 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:12.800511 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 08:01:12.800617 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:12.800571 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls podName:55f885a2-8fd2-4059-a3a5-988292116572 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:28.800554505 +0000 UTC m=+66.220299294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls") pod "dns-default-896pl" (UID: "55f885a2-8fd2-4059-a3a5-988292116572") : secret "dns-default-metrics-tls" not found Jul 15 08:01:13.323373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:13.323326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" event={"ID":"f54e03d5-92ec-4a3e-a850-d70351483a37","Type":"ContainerStarted","Data":"2f9d8b6edc6b5504f2f81ae97f95ee3820ee2778182537bfdb823d397bcca15c"} Jul 15 08:01:13.713132 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:13.713051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:13.713223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.713229 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.713301 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.713283632 +0000 UTC m=+55.133028431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.713365 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:13.713365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.713441 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls podName:7813420f-c6bb-4140-b981-6a53eae98b4c nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.713425637 +0000 UTC m=+55.133170421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls") pod "cluster-samples-operator-65db858894-5qffs" (UID: "7813420f-c6bb-4140-b981-6a53eae98b4c") : secret "samples-operator-tls" not found Jul 15 08:01:13.713535 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.713486 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.713477376 +0000 UTC m=+55.133222157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:13.814713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:13.814677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:13.814882 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.814838 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:13.814928 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:13.814913 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.814898966 +0000 UTC m=+55.234643747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:15.333709 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.333642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" event={"ID":"f8d7f2ca-b85d-4084-9138-4d71c041be8a","Type":"ContainerStarted","Data":"fb1f62dedc581d1e390581790258fb29fb23d26e2544a47c19ee0d18f3ae5bfb"} Jul 15 08:01:15.335884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.335859 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" event={"ID":"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb","Type":"ContainerStarted","Data":"a98c252f0d2f2260813f19ee11e9bfba452061cec44c00356295a6962ca21b4a"} Jul 15 08:01:15.339253 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.339229 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" event={"ID":"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b","Type":"ContainerStarted","Data":"15900f624af5e0d547e4a0803cc3576544368ec3ffb08721651417c03c7a25b3"} Jul 15 08:01:15.342269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.342248 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/0.log" Jul 15 08:01:15.342345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.342286 2565 generic.go:356] "Generic (PLEG): container finished" podID="34bf1e10-21d6-4ea5-8abe-30009956d7ce" containerID="b69121687792522210fe2599070080b28544717a3e584510beadfd9edea4ce5a" exitCode=255 Jul 15 08:01:15.342397 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.342352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r542h" event={"ID":"34bf1e10-21d6-4ea5-8abe-30009956d7ce","Type":"ContainerDied","Data":"b69121687792522210fe2599070080b28544717a3e584510beadfd9edea4ce5a"} Jul 15 08:01:15.342641 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.342625 2565 scope.go:117] "RemoveContainer" containerID="b69121687792522210fe2599070080b28544717a3e584510beadfd9edea4ce5a" Jul 15 08:01:15.344521 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.344493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" event={"ID":"7f3d67b1-ae4c-4494-b8be-54edcc5977c9","Type":"ContainerStarted","Data":"4dcb344af83cdfb6499986acce7c8c8459d0b6513610999f0d2b6544f7547e6d"} Jul 15 08:01:15.353494 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.353422 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" podStartSLOduration=1.609661149 podStartE2EDuration="6.353393681s" totalImagesPullingTime="4.743732532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:15.353192919 +0000 UTC m=+52.772937722" watchObservedRunningTime="2026-07-15 08:01:15.353393681 +0000 UTC m=+52.773138485" Jul 15 08:01:15.370433 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.370025 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-5kfn6" podStartSLOduration=1.819127184 podStartE2EDuration="6.370007164s" totalImagesPullingTime="4.55087998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:15.36952987 +0000 UTC m=+52.789274675" watchObservedRunningTime="2026-07-15 08:01:15.370007164 +0000 UTC m=+52.789751952" Jul 15 08:01:15.420625 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.420583 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" podStartSLOduration=1.914897679 podStartE2EDuration="6.42056693s" totalImagesPullingTime="4.505669251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:15.388633071 +0000 UTC m=+52.808377874" watchObservedRunningTime="2026-07-15 08:01:15.42056693 +0000 UTC m=+52.840311743" Jul 15 08:01:15.444038 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:15.443241 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" podStartSLOduration=1.910623857 podStartE2EDuration="6.443222169s" totalImagesPullingTime="4.532598312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:15.441151754 +0000 UTC m=+52.860896570" watchObservedRunningTime="2026-07-15 08:01:15.443222169 +0000 UTC m=+52.862966978" Jul 15 08:01:16.351423 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.350980 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:01:16.352162 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.352106 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/0.log" Jul 15 08:01:16.352250 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.352192 2565 generic.go:356] "Generic (PLEG): container finished" podID="34bf1e10-21d6-4ea5-8abe-30009956d7ce" containerID="9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672" exitCode=255 Jul 15 08:01:16.352308 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.352275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r542h" event={"ID":"34bf1e10-21d6-4ea5-8abe-30009956d7ce","Type":"ContainerDied","Data":"9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672"} Jul 15 08:01:16.352374 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.352358 2565 scope.go:117] "RemoveContainer" containerID="b69121687792522210fe2599070080b28544717a3e584510beadfd9edea4ce5a" Jul 15 08:01:16.352676 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.352585 2565 scope.go:117] "RemoveContainer" containerID="9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672" Jul 15 08:01:16.352791 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:16.352766 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r542h_openshift-console-operator(34bf1e10-21d6-4ea5-8abe-30009956d7ce)\"" pod="openshift-console-operator/console-operator-7866497494-r542h" podUID="34bf1e10-21d6-4ea5-8abe-30009956d7ce" Jul 15 08:01:16.697558 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.697416 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5fcm7"] Jul 15 08:01:16.702099 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.702065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.704667 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.704644 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 08:01:16.704795 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.704776 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2tcw\"" Jul 15 08:01:16.705005 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.704987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 08:01:16.711030 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.710977 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5fcm7"] Jul 15 08:01:16.745559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.745511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944e658e-f32c-449b-9be1-1f7d5e922c18-crio-socket\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.745743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.745646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e658e-f32c-449b-9be1-1f7d5e922c18-data-volume\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.745743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.745723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6524\" (UniqueName: \"kubernetes.io/projected/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-api-access-h6524\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.745869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.745817 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.745926 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.745911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e658e-f32c-449b-9be1-1f7d5e922c18-data-volume\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6524\" (UniqueName: \"kubernetes.io/projected/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-api-access-h6524\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944e658e-f32c-449b-9be1-1f7d5e922c18-crio-socket\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:16.846823 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:16.846844 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944e658e-f32c-449b-9be1-1f7d5e922c18-crio-socket\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.846860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e658e-f32c-449b-9be1-1f7d5e922c18-data-volume\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:16.846894 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.34687406 +0000 UTC m=+54.766618845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : configmap "kube-rbac-proxy" not found Jul 15 08:01:16.846925 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:16.846918 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:17.346905787 +0000 UTC m=+54.766650575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:16.858179 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:16.858151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6524\" (UniqueName: \"kubernetes.io/projected/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-api-access-h6524\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:17.351297 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.351258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:17.351509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.351381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:17.351509 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.351382 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:17.351892 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.351534 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:18.351514783 +0000 UTC m=+55.771259571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : configmap "kube-rbac-proxy" not found Jul 15 08:01:17.351892 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.351471 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:17.351892 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.351576 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:18.351567324 +0000 UTC m=+55.771312108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:17.362380 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.362342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:01:17.362755 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.362730 2565 scope.go:117] "RemoveContainer" containerID="9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672" Jul 15 08:01:17.362914 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.362896 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r542h_openshift-console-operator(34bf1e10-21d6-4ea5-8abe-30009956d7ce)\"" pod="openshift-console-operator/console-operator-7866497494-r542h" podUID="34bf1e10-21d6-4ea5-8abe-30009956d7ce" Jul 15 08:01:17.754582 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.754543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:17.754764 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.754648 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:17.754764 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.754717 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:17.754764 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.754755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:17.754932 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.754806 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.754783021 +0000 UTC m=+63.174527812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:17.754932 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.754878 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.754862243 +0000 UTC m=+63.174607039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:17.755047 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.754942 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 08:01:17.755047 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.754973 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls podName:7813420f-c6bb-4140-b981-6a53eae98b4c nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.754962664 +0000 UTC m=+63.174707448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls") pod "cluster-samples-operator-65db858894-5qffs" (UID: "7813420f-c6bb-4140-b981-6a53eae98b4c") : secret "samples-operator-tls" not found Jul 15 08:01:17.856130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:17.856096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:17.856296 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.856237 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:17.856354 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:17.856303 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:25.85628941 +0000 UTC m=+63.276034191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:18.102867 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:18.102791 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jhvd_a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6/dns-node-resolver/0.log" Jul 15 08:01:18.362024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:18.361926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:18.362385 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:18.362038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:18.362385 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:18.362059 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:18.362385 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:18.362127 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:20.362106784 +0000 UTC m=+57.781851564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : configmap "kube-rbac-proxy" not found Jul 15 08:01:18.362385 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:18.362185 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:18.362385 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:18.362252 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:20.362236445 +0000 UTC m=+57.781981243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:18.366720 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:18.366693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" event={"ID":"f54e03d5-92ec-4a3e-a850-d70351483a37","Type":"ContainerStarted","Data":"cf0e200674084af4d8b7e65a488059c9433fef562d99de43f692c0eb30f82d35"} Jul 15 08:01:18.391039 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:18.390997 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-f2rdl" podStartSLOduration=2.197516878 podStartE2EDuration="7.390980667s" totalImagesPullingTime="5.193463789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:18.389977487 +0000 UTC m=+55.809722291" watchObservedRunningTime="2026-07-15 08:01:18.390980667 +0000 UTC m=+55.810725471" Jul 15 08:01:19.303301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:19.303276 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmdh8_1d4cff77-0eaf-4210-9839-c8a3cbe90021/node-ca/0.log" Jul 15 08:01:19.905337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:19.905306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8s45p_6a0b8373-62f9-4798-a412-b5ba19e87b42/serve-healthcheck-canary/0.log" Jul 15 08:01:20.379692 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:20.379660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:20.379872 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:20.379767 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:20.379872 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:20.379827 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:24.379814062 +0000 UTC m=+61.799558843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : configmap "kube-rbac-proxy" not found Jul 15 08:01:20.379872 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:20.379847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:20.379996 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:20.379946 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:20.379996 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:20.379971 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:24.379963681 +0000 UTC m=+61.799708462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:20.533687 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:20.533656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:20.533829 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:20.533712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:20.534023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:20.534011 2565 scope.go:117] "RemoveContainer" containerID="9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672" Jul 15 08:01:20.534164 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:20.534147 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r542h_openshift-console-operator(34bf1e10-21d6-4ea5-8abe-30009956d7ce)\"" pod="openshift-console-operator/console-operator-7866497494-r542h" podUID="34bf1e10-21d6-4ea5-8abe-30009956d7ce" Jul 15 08:01:23.264653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:23.264622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m7n97" Jul 15 08:01:24.413368 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:24.413328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:24.413811 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:24.413452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:24.413811 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:24.413478 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 08:01:24.413811 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:24.413534 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:32.413519075 +0000 UTC m=+69.833263857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : configmap "kube-rbac-proxy" not found Jul 15 08:01:24.413811 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:24.413591 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:24.413811 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:24.413656 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:32.413643827 +0000 UTC m=+69.833388607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:25.825871 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.825835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:25.826257 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.825890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:25.826257 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.825936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:25.826257 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:25.826013 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:41.825994674 +0000 UTC m=+79.245739455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : configmap references non-existent config key: service-ca.crt Jul 15 08:01:25.826257 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:25.826040 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 08:01:25.826257 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:25.826099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs podName:55aeea10-575f-4ca6-b899-02f564240e75 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:41.826084068 +0000 UTC m=+79.245828850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs") pod "router-default-6f8844f56-rsmbf" (UID: "55aeea10-575f-4ca6-b899-02f564240e75") : secret "router-metrics-certs-default" not found Jul 15 08:01:25.828279 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.828258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7813420f-c6bb-4140-b981-6a53eae98b4c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-5qffs\" (UID: \"7813420f-c6bb-4140-b981-6a53eae98b4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:25.927142 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.927109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:25.927297 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:25.927242 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:25.927344 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:25.927307 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls podName:fb48dc08-fe83-496b-96ad-9889a57b224f nodeName:}" failed. No retries permitted until 2026-07-15 08:01:41.927290928 +0000 UTC m=+79.347035721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8vd7w" (UID: "fb48dc08-fe83-496b-96ad-9889a57b224f") : secret "cluster-monitoring-operator-tls" not found Jul 15 08:01:25.938092 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.938066 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2hc7p\"" Jul 15 08:01:25.946830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:25.946812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" Jul 15 08:01:26.067205 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:26.067176 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs"] Jul 15 08:01:26.391749 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:26.391666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" event={"ID":"7813420f-c6bb-4140-b981-6a53eae98b4c","Type":"ContainerStarted","Data":"abfc36f98f173cf8509522bcc4a9b6c283e9b18899a33d1b3c778ca438e03b8f"} Jul 15 08:01:28.399555 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.399519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" event={"ID":"7813420f-c6bb-4140-b981-6a53eae98b4c","Type":"ContainerStarted","Data":"d3417cac90b4322f09aa43ac6a1c631af5e9e61abf55a0bfc745053c2f77e2b2"} Jul 15 08:01:28.399920 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.399572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" event={"ID":"7813420f-c6bb-4140-b981-6a53eae98b4c","Type":"ContainerStarted","Data":"f78c003a644c0d04961177f47b6cee7442f7dda02bc80834c54ef1eff969555d"} Jul 15 08:01:28.417250 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.417207 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-5qffs" podStartSLOduration=17.352430271 podStartE2EDuration="19.417191184s" totalImagesPullingTime="2.064760913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:28.416653365 +0000 UTC m=+65.836398168" watchObservedRunningTime="2026-07-15 08:01:28.417191184 +0000 UTC m=+65.836935984" Jul 15 08:01:28.753231 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.753195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:28.755507 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.755480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"image-registry-66dcc4df8-htzcn\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:28.853850 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.853813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:01:28.853995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.853914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:01:28.853995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.853965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:01:28.856184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.856155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f885a2-8fd2-4059-a3a5-988292116572-metrics-tls\") pod \"dns-default-896pl\" (UID: \"55f885a2-8fd2-4059-a3a5-988292116572\") " pod="openshift-dns/dns-default-896pl" Jul 15 08:01:28.856463 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.856446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 08:01:28.856515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.856501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 08:01:28.864427 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:28.864396 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 08:01:28.864511 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:28.864495 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs podName:44438dfa-d58b-48f6-9002-0bbd27cf84c5 nodeName:}" failed. No retries permitted until 2026-07-15 08:02:32.864465023 +0000 UTC m=+130.284209820 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs") pod "network-metrics-daemon-jh48p" (UID: "44438dfa-d58b-48f6-9002-0bbd27cf84c5") : secret "metrics-daemon-secret" not found Jul 15 08:01:28.866250 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.866225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb9d64a-d1b4-4feb-a041-4de44d59beaa-original-pull-secret\") pod \"global-pull-secret-syncer-f2s78\" (UID: \"bbb9d64a-d1b4-4feb-a041-4de44d59beaa\") " pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:01:28.928598 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.928555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2s78" Jul 15 08:01:28.954908 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.954875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:01:28.957262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.957240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrxm\" (UniqueName: \"kubernetes.io/projected/616fe1a6-17ce-4a60-995c-b047e24d6338-kube-api-access-csrxm\") pod \"network-check-target-ptzbf\" (UID: \"616fe1a6-17ce-4a60-995c-b047e24d6338\") " pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:01:28.982117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.981964 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wb89\"" Jul 15 08:01:28.990160 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:28.990130 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:29.054084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.053783 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2s78"] Jul 15 08:01:29.059449 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.058738 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8tgdm\"" Jul 15 08:01:29.065798 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.065772 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-896pl" Jul 15 08:01:29.148828 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.148795 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:01:29.151904 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:29.151864 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfe29f9_411a_4def_9aaf_41cbea161e06.slice/crio-967d8a688217034da23b375a7be06bc92ff6bc6c7631781e674e1bc376e76990 WatchSource:0}: Error finding container 967d8a688217034da23b375a7be06bc92ff6bc6c7631781e674e1bc376e76990: Status 404 returned error can't find the container with id 967d8a688217034da23b375a7be06bc92ff6bc6c7631781e674e1bc376e76990 Jul 15 08:01:29.208929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.208907 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-896pl"] Jul 15 08:01:29.216453 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:29.216425 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f885a2_8fd2_4059_a3a5_988292116572.slice/crio-5f6aa903562ee40ac9627e4fd7c0fe1916331248ec00167b871242e8a718985e WatchSource:0}: Error finding container 5f6aa903562ee40ac9627e4fd7c0fe1916331248ec00167b871242e8a718985e: Status 404 returned error can't find the container with id 5f6aa903562ee40ac9627e4fd7c0fe1916331248ec00167b871242e8a718985e Jul 15 08:01:29.236305 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.236287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5x8qz\"" Jul 15 08:01:29.244236 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.244215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:01:29.374000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.373979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ptzbf"] Jul 15 08:01:29.377021 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:29.376996 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616fe1a6_17ce_4a60_995c_b047e24d6338.slice/crio-64b34cc662a5eab3457224586a1b3b249561aa479ca6240ccfa3f9a8d7403925 WatchSource:0}: Error finding container 64b34cc662a5eab3457224586a1b3b249561aa479ca6240ccfa3f9a8d7403925: Status 404 returned error can't find the container with id 64b34cc662a5eab3457224586a1b3b249561aa479ca6240ccfa3f9a8d7403925 Jul 15 08:01:29.405170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.405139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" event={"ID":"5dfe29f9-411a-4def-9aaf-41cbea161e06","Type":"ContainerStarted","Data":"4eb0a7726841155263e4f72e69dc118842fbdd21811dde42788a50f9c1e384d1"} Jul 15 08:01:29.405522 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.405188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" event={"ID":"5dfe29f9-411a-4def-9aaf-41cbea161e06","Type":"ContainerStarted","Data":"967d8a688217034da23b375a7be06bc92ff6bc6c7631781e674e1bc376e76990"} Jul 15 08:01:29.405522 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.405228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:29.406322 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.406297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ptzbf" event={"ID":"616fe1a6-17ce-4a60-995c-b047e24d6338","Type":"ContainerStarted","Data":"64b34cc662a5eab3457224586a1b3b249561aa479ca6240ccfa3f9a8d7403925"} Jul 15 08:01:29.407432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.407387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-896pl" event={"ID":"55f885a2-8fd2-4059-a3a5-988292116572","Type":"ContainerStarted","Data":"5f6aa903562ee40ac9627e4fd7c0fe1916331248ec00167b871242e8a718985e"} Jul 15 08:01:29.408989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.408964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2s78" event={"ID":"bbb9d64a-d1b4-4feb-a041-4de44d59beaa","Type":"ContainerStarted","Data":"7c6b995bf5c72ea0bfd72d3d2296b52d92aac0049e5499b69f89849c4d419ef2"} Jul 15 08:01:29.425192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:29.425152 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" podStartSLOduration=66.42513879 podStartE2EDuration="1m6.42513879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:29.423620546 +0000 UTC m=+66.843365349" watchObservedRunningTime="2026-07-15 08:01:29.42513879 +0000 UTC m=+66.844883592" Jul 15 08:01:30.415160 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:30.415123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ptzbf" event={"ID":"616fe1a6-17ce-4a60-995c-b047e24d6338","Type":"ContainerStarted","Data":"8a638a44c917b9046a47d1b96a5a91d38053af4b98a377e4ea02a1578305805f"} Jul 15 08:01:30.415721 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:30.415443 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:01:30.441935 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:30.441885 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ptzbf" podStartSLOduration=67.441868553 podStartE2EDuration="1m7.441868553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:30.440498701 +0000 UTC m=+67.860243506" watchObservedRunningTime="2026-07-15 08:01:30.441868553 +0000 UTC m=+67.861613354" Jul 15 08:01:31.421845 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:31.421735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-896pl" event={"ID":"55f885a2-8fd2-4059-a3a5-988292116572","Type":"ContainerStarted","Data":"70a4320b3bfc6fc08eb79fd817912d57f2104d17a8e504b7f59b92ba85b9abb9"} Jul 15 08:01:31.421845 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:31.421814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-896pl" event={"ID":"55f885a2-8fd2-4059-a3a5-988292116572","Type":"ContainerStarted","Data":"0fd221c00856d1189b14d13fdb3307bad9f7eebc5a82dca81ca672c92781397b"} Jul 15 08:01:31.422259 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:31.421852 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-896pl" Jul 15 08:01:31.442932 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:31.442870 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-896pl" podStartSLOduration=33.611098967 podStartE2EDuration="35.442849219s" totalImagesPullingTime="1.831750252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:31.442201681 +0000 UTC m=+68.861946505" watchObservedRunningTime="2026-07-15 08:01:31.442849219 +0000 UTC m=+68.862594024" Jul 15 08:01:32.488493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:32.488445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:32.488965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:32.488536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:32.488965 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:32.488653 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 08:01:32.488965 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:32.488714 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls podName:944e658e-f32c-449b-9be1-1f7d5e922c18 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:48.488696442 +0000 UTC m=+85.908441227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fcm7" (UID: "944e658e-f32c-449b-9be1-1f7d5e922c18") : secret "insights-runtime-extractor-tls" not found Jul 15 08:01:32.489141 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:32.489101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944e658e-f32c-449b-9be1-1f7d5e922c18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:35.115736 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.115704 2565 scope.go:117] "RemoveContainer" containerID="9de0e5cbde76a0db41a976182d653fce505d91396aa591799587f5e34a7fc672" Jul 15 08:01:35.440202 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.440104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2s78" event={"ID":"bbb9d64a-d1b4-4feb-a041-4de44d59beaa","Type":"ContainerStarted","Data":"24aa6f893ad803ad74965a152b324a91e9dac43e2afc44e7871c6c84c2fb766a"} Jul 15 08:01:35.441840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.441820 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:01:35.441957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.441867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r542h" event={"ID":"34bf1e10-21d6-4ea5-8abe-30009956d7ce","Type":"ContainerStarted","Data":"f450bdfb489c0cf8ce0fa63b97fb46c6ead06c73904ac43b4fea60b3ae4be9b7"} Jul 15 08:01:35.442135 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.442117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:35.457005 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.456960 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f2s78" podStartSLOduration=66.952225172 podStartE2EDuration="1m12.456944887s" totalImagesPullingTime="5.504719715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:35.45669607 +0000 UTC m=+72.876440873" watchObservedRunningTime="2026-07-15 08:01:35.456944887 +0000 UTC m=+72.876689692" Jul 15 08:01:35.472727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.472689 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-r542h" podStartSLOduration=21.068463981 podStartE2EDuration="25.47267671s" totalImagesPullingTime="4.404212729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:35.472073158 +0000 UTC m=+72.891817961" watchObservedRunningTime="2026-07-15 08:01:35.47267671 +0000 UTC m=+72.892421513" Jul 15 08:01:35.985041 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:35.985009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-r542h" Jul 15 08:01:41.430337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.430304 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-896pl" Jul 15 08:01:41.870299 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.870258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:41.870503 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.870344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:41.870922 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.870897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55aeea10-575f-4ca6-b899-02f564240e75-service-ca-bundle\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:41.872597 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.872577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55aeea10-575f-4ca6-b899-02f564240e75-metrics-certs\") pod \"router-default-6f8844f56-rsmbf\" (UID: \"55aeea10-575f-4ca6-b899-02f564240e75\") " pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:41.972733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.972694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:41.975117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:41.975086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb48dc08-fe83-496b-96ad-9889a57b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8vd7w\" (UID: \"fb48dc08-fe83-496b-96ad-9889a57b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:42.157505 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.157393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cwj6j\"" Jul 15 08:01:42.166211 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.166179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:42.248917 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.248881 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hgl68\"" Jul 15 08:01:42.256962 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.256937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" Jul 15 08:01:42.300219 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.300190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8844f56-rsmbf"] Jul 15 08:01:42.303656 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:42.303626 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aeea10_575f_4ca6_b899_02f564240e75.slice/crio-745f9610e1862579f3bc6510090051c89954ad99803dae896fbfaad15ec75ef4 WatchSource:0}: Error finding container 745f9610e1862579f3bc6510090051c89954ad99803dae896fbfaad15ec75ef4: Status 404 returned error can't find the container with id 745f9610e1862579f3bc6510090051c89954ad99803dae896fbfaad15ec75ef4 Jul 15 08:01:42.391559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.391532 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w"] Jul 15 08:01:42.394623 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:42.394599 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb48dc08_fe83_496b_96ad_9889a57b224f.slice/crio-bb82bfa090e2ae1595c518fa57f4b1d0fe58f2a534fefae42727c2c3f2eb31d1 WatchSource:0}: Error finding container bb82bfa090e2ae1595c518fa57f4b1d0fe58f2a534fefae42727c2c3f2eb31d1: Status 404 returned error can't find the container with id bb82bfa090e2ae1595c518fa57f4b1d0fe58f2a534fefae42727c2c3f2eb31d1 Jul 15 08:01:42.468116 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.468033 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" event={"ID":"fb48dc08-fe83-496b-96ad-9889a57b224f","Type":"ContainerStarted","Data":"bb82bfa090e2ae1595c518fa57f4b1d0fe58f2a534fefae42727c2c3f2eb31d1"} Jul 15 08:01:42.469555 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.469525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8844f56-rsmbf" event={"ID":"55aeea10-575f-4ca6-b899-02f564240e75","Type":"ContainerStarted","Data":"40692b004439931b71a67979aedb3c575b3b31859374f13348fd26267c85b5bb"} Jul 15 08:01:42.469682 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.469563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8844f56-rsmbf" event={"ID":"55aeea10-575f-4ca6-b899-02f564240e75","Type":"ContainerStarted","Data":"745f9610e1862579f3bc6510090051c89954ad99803dae896fbfaad15ec75ef4"} Jul 15 08:01:42.491593 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:42.491548 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f8844f56-rsmbf" podStartSLOduration=33.491533673 podStartE2EDuration="33.491533673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:42.49123928 +0000 UTC m=+79.910984109" watchObservedRunningTime="2026-07-15 08:01:42.491533673 +0000 UTC m=+79.911278473" Jul 15 08:01:43.167124 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:43.167087 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:43.170904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:43.170880 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:43.473025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:43.472955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:43.474290 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:43.474266 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f8844f56-rsmbf" Jul 15 08:01:44.478499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.478455 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" event={"ID":"fb48dc08-fe83-496b-96ad-9889a57b224f","Type":"ContainerStarted","Data":"7f965d342875348ed40c157e1cb5b77dfce70ced5d7dc2dacd2c1471c71dc6d2"} Jul 15 08:01:44.496707 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.496657 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8vd7w" podStartSLOduration=32.596074952 podStartE2EDuration="34.49663594s" totalImagesPullingTime="1.900560988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:44.496227922 +0000 UTC m=+81.915972726" watchObservedRunningTime="2026-07-15 08:01:44.49663594 +0000 UTC m=+81.916380743" Jul 15 08:01:44.937580 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.937550 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b"] Jul 15 08:01:44.940779 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.940763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:44.947218 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.947196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 08:01:44.947389 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.947369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l5j5j\"" Jul 15 08:01:44.956083 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.956061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b"] Jul 15 08:01:44.999287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:44.999260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-c4q4b\" (UID: \"bb053f9b-e5bf-422f-98ce-97ccaf2557cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:45.100343 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:45.100312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-c4q4b\" (UID: \"bb053f9b-e5bf-422f-98ce-97ccaf2557cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:45.100481 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:45.100432 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 15 08:01:45.100538 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:45.100482 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates podName:bb053f9b-e5bf-422f-98ce-97ccaf2557cf nodeName:}" failed. No retries permitted until 2026-07-15 08:01:45.600469568 +0000 UTC m=+83.020214348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-c4q4b" (UID: "bb053f9b-e5bf-422f-98ce-97ccaf2557cf") : secret "prometheus-operator-admission-webhook-tls" not found Jul 15 08:01:45.604988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:45.604867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-c4q4b\" (UID: \"bb053f9b-e5bf-422f-98ce-97ccaf2557cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:45.605360 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:45.605009 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 15 08:01:45.605360 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:45.605093 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates podName:bb053f9b-e5bf-422f-98ce-97ccaf2557cf nodeName:}" failed. No retries permitted until 2026-07-15 08:01:46.605074044 +0000 UTC m=+84.024818842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-c4q4b" (UID: "bb053f9b-e5bf-422f-98ce-97ccaf2557cf") : secret "prometheus-operator-admission-webhook-tls" not found Jul 15 08:01:46.611537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:46.611491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-c4q4b\" (UID: \"bb053f9b-e5bf-422f-98ce-97ccaf2557cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:46.613828 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:46.613809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb053f9b-e5bf-422f-98ce-97ccaf2557cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-c4q4b\" (UID: \"bb053f9b-e5bf-422f-98ce-97ccaf2557cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:46.749463 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:46.749424 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:46.908915 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:46.908821 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b"] Jul 15 08:01:46.911698 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:46.911672 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb053f9b_e5bf_422f_98ce_97ccaf2557cf.slice/crio-ea4850a08f20a00fc75202311b96955425938d80093bff37abaa0f2ab892902c WatchSource:0}: Error finding container ea4850a08f20a00fc75202311b96955425938d80093bff37abaa0f2ab892902c: Status 404 returned error can't find the container with id ea4850a08f20a00fc75202311b96955425938d80093bff37abaa0f2ab892902c Jul 15 08:01:47.490860 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:47.490829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" event={"ID":"bb053f9b-e5bf-422f-98ce-97ccaf2557cf","Type":"ContainerStarted","Data":"ea4850a08f20a00fc75202311b96955425938d80093bff37abaa0f2ab892902c"} Jul 15 08:01:48.527022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.526980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:48.529271 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.529247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944e658e-f32c-449b-9be1-1f7d5e922c18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fcm7\" (UID: \"944e658e-f32c-449b-9be1-1f7d5e922c18\") " pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:48.817925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.817849 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2tcw\"" Jul 15 08:01:48.825698 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.825680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5fcm7" Jul 15 08:01:48.971505 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.971432 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5fcm7"] Jul 15 08:01:48.974294 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:48.974270 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944e658e_f32c_449b_9be1_1f7d5e922c18.slice/crio-17ccddebdfa080dae62636bb449e5483e723d55e2afc077535eada176eb67381 WatchSource:0}: Error finding container 17ccddebdfa080dae62636bb449e5483e723d55e2afc077535eada176eb67381: Status 404 returned error can't find the container with id 17ccddebdfa080dae62636bb449e5483e723d55e2afc077535eada176eb67381 Jul 15 08:01:48.994209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.994186 2565 patch_prober.go:28] interesting pod/image-registry-66dcc4df8-htzcn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 08:01:48.994312 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:48.994229 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 08:01:49.109120 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.109047 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:01:49.137031 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.137004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.142155 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.142125 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 08:01:49.142308 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.142215 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 08:01:49.142308 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.142244 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 08:01:49.142308 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.142269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 08:01:49.143046 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.143022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 08:01:49.143169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.143146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 08:01:49.143253 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.143236 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kcpks\"" Jul 15 08:01:49.143345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.143327 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 08:01:49.145262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.145245 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:01:49.232625 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.232625 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrthz\" (UniqueName: \"kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.232827 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.232827 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.232827 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.232827 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.232823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333553 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.333712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.333704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrthz\" (UniqueName: \"kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.334391 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.334366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.334542 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.334519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.334614 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.334519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.336009 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.335972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.336098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.336022 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.341610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.341589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrthz\" (UniqueName: \"kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz\") pod \"console-55fcc7bb9-jnb6d\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.450656 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.450559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:49.499863 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.499829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fcm7" event={"ID":"944e658e-f32c-449b-9be1-1f7d5e922c18","Type":"ContainerStarted","Data":"17ccddebdfa080dae62636bb449e5483e723d55e2afc077535eada176eb67381"} Jul 15 08:01:49.501514 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.501488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" event={"ID":"bb053f9b-e5bf-422f-98ce-97ccaf2557cf","Type":"ContainerStarted","Data":"85b942ef7fb69df431d2d16ffd242096decb4bac42fad063d8fc172b7cd27e35"} Jul 15 08:01:49.501709 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.501675 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:49.506374 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.506353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" Jul 15 08:01:49.525701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.525549 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-c4q4b" podStartSLOduration=3.774738438 podStartE2EDuration="5.525527688s" totalImagesPullingTime="1.75078925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:49.524666045 +0000 UTC m=+86.944410849" watchObservedRunningTime="2026-07-15 08:01:49.525527688 +0000 UTC m=+86.945272489" Jul 15 08:01:49.595286 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:49.595229 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:01:49.597846 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:49.597815 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90cf80b_b0ef_4539_ba23_14bd52451849.slice/crio-9e00676fb0f956ad86a4ee0227b3728bb3090e1a4f6bb9c836d54b5c0b799949 WatchSource:0}: Error finding container 9e00676fb0f956ad86a4ee0227b3728bb3090e1a4f6bb9c836d54b5c0b799949: Status 404 returned error can't find the container with id 9e00676fb0f956ad86a4ee0227b3728bb3090e1a4f6bb9c836d54b5c0b799949 Jul 15 08:01:50.421173 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:50.421140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:01:50.507429 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:50.507379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fcc7bb9-jnb6d" event={"ID":"a90cf80b-b0ef-4539-ba23-14bd52451849","Type":"ContainerStarted","Data":"9e00676fb0f956ad86a4ee0227b3728bb3090e1a4f6bb9c836d54b5c0b799949"} Jul 15 08:01:56.534740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:56.534695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fcm7" event={"ID":"944e658e-f32c-449b-9be1-1f7d5e922c18","Type":"ContainerStarted","Data":"99674591cc480b614e9a3073b3e8c791f7317f106a410d877ccce6bd7c3254b3"} Jul 15 08:01:56.536771 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:56.536724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fcc7bb9-jnb6d" event={"ID":"a90cf80b-b0ef-4539-ba23-14bd52451849","Type":"ContainerStarted","Data":"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2"} Jul 15 08:01:56.555387 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:56.555287 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55fcc7bb9-jnb6d" podStartSLOduration=1.556443277 podStartE2EDuration="7.555269403s" totalImagesPullingTime="5.998826126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:01:56.553901371 +0000 UTC m=+93.973646176" watchObservedRunningTime="2026-07-15 08:01:56.555269403 +0000 UTC m=+93.975014207" Jul 15 08:01:57.448931 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.448902 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c6qsr"] Jul 15 08:01:57.452700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.452673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.455334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.455310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fqpfw\"" Jul 15 08:01:57.455515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.455313 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 08:01:57.455729 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.455696 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 08:01:57.455879 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.455860 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 08:01:57.455955 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.455897 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 08:01:57.508084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-root\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-textfile\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-sys\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-wtmp\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508605 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-metrics-client-ca\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.508605 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.508284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8gh\" (UniqueName: \"kubernetes.io/projected/6b82e419-9f91-4efb-bf58-83caf297304a-kube-api-access-4h8gh\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-root\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-textfile\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-root\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-sys\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-wtmp\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-metrics-client-ca\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8gh\" (UniqueName: \"kubernetes.io/projected/6b82e419-9f91-4efb-bf58-83caf297304a-kube-api-access-4h8gh\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.609997 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.609921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.610476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.610010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-textfile\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.610476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.610210 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.610580 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.610546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b82e419-9f91-4efb-bf58-83caf297304a-metrics-client-ca\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.610631 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.610597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-sys\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.610688 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:57.610636 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 08:01:57.610727 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:57.610712 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls podName:6b82e419-9f91-4efb-bf58-83caf297304a nodeName:}" failed. No retries permitted until 2026-07-15 08:01:58.110692164 +0000 UTC m=+95.530436961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls") pod "node-exporter-c6qsr" (UID: "6b82e419-9f91-4efb-bf58-83caf297304a") : secret "node-exporter-tls" not found Jul 15 08:01:57.610783 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.610645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-wtmp\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.612717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.612697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:57.620354 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:57.620322 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8gh\" (UniqueName: \"kubernetes.io/projected/6b82e419-9f91-4efb-bf58-83caf297304a-kube-api-access-4h8gh\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:58.114170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.114122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:58.116760 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.116733 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b82e419-9f91-4efb-bf58-83caf297304a-node-exporter-tls\") pod \"node-exporter-c6qsr\" (UID: \"6b82e419-9f91-4efb-bf58-83caf297304a\") " pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:58.363745 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.363709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c6qsr" Jul 15 08:01:58.443930 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.443900 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:01:58.449843 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.449822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.452014 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.451987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 08:01:58.452145 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452108 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 08:01:58.452220 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452452 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452715 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452715 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452777 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.452955 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wfqqv\"" Jul 15 08:01:58.453262 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.453039 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 08:01:58.464257 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.464235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:01:58.517385 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.517765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.518106 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.518106 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.517794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkblb\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.618557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.619023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.618989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkblb\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.620088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.619830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.620941 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:01:58.620437 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle podName:62826120-d9a2-40b8-ba83-399605b51676 nodeName:}" failed. No retries permitted until 2026-07-15 08:01:59.120394058 +0000 UTC m=+96.540138844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "62826120-d9a2-40b8-ba83-399605b51676") : configmap references non-existent config key: ca-bundle.crt Jul 15 08:01:58.621851 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.621830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.622568 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.622526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.625943 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.625901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.626701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.626636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.627452 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.627389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:58.629717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:58.629695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkblb\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:59.124496 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.123822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:59.124694 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.124670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:59.361995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.361961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:01:59.413953 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:59.413923 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b82e419_9f91_4efb_bf58_83caf297304a.slice/crio-d8088413b010a4474ab9f83d2c05efd0aa5faa4c58f19db042124b74cc5fcef6 WatchSource:0}: Error finding container d8088413b010a4474ab9f83d2c05efd0aa5faa4c58f19db042124b74cc5fcef6: Status 404 returned error can't find the container with id d8088413b010a4474ab9f83d2c05efd0aa5faa4c58f19db042124b74cc5fcef6 Jul 15 08:01:59.452222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.451530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:59.452222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.451619 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:59.457306 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.457268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:59.549294 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.549249 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6qsr" event={"ID":"6b82e419-9f91-4efb-bf58-83caf297304a","Type":"ContainerStarted","Data":"d8088413b010a4474ab9f83d2c05efd0aa5faa4c58f19db042124b74cc5fcef6"} Jul 15 08:01:59.551282 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.551254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fcm7" event={"ID":"944e658e-f32c-449b-9be1-1f7d5e922c18","Type":"ContainerStarted","Data":"887bffc7bbbf43b2bd1ed4c476ef2ccefb0837304cfb2c772aeea68c9f9875f7"} Jul 15 08:01:59.555115 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.555091 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:01:59.559937 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.559916 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:01:59.561551 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:01:59.561528 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62826120_d9a2_40b8_ba83_399605b51676.slice/crio-cfc0bdff08386ebae42e5ccdf839ffef8b3c0b07b5f3dc17a21318eddbe9fad0 WatchSource:0}: Error finding container cfc0bdff08386ebae42e5ccdf839ffef8b3c0b07b5f3dc17a21318eddbe9fad0: Status 404 returned error can't find the container with id cfc0bdff08386ebae42e5ccdf839ffef8b3c0b07b5f3dc17a21318eddbe9fad0 Jul 15 08:01:59.592803 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.592776 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d45c569c-wtk76"] Jul 15 08:01:59.596630 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.596609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.603199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.602063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 08:01:59.603199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.602335 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q7zwq\"" Jul 15 08:01:59.603199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.602582 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fob3j8j1qgijp\"" Jul 15 08:01:59.603199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.602848 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 08:01:59.603199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.603101 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 08:01:59.603551 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.603313 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 08:01:59.603551 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.603433 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 08:01:59.610359 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.610330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d45c569c-wtk76"] Jul 15 08:01:59.628868 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.628840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629246 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.628895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de593aa0-7112-4d96-ba10-5725750c3072-metrics-client-ca\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629246 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-grpc-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629246 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkln\" (UniqueName: \"kubernetes.io/projected/de593aa0-7112-4d96-ba10-5725750c3072-kube-api-access-swkln\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629464 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.629464 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.629421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.730731 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.730892 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.730892 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.730892 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de593aa0-7112-4d96-ba10-5725750c3072-metrics-client-ca\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.730892 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-grpc-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.731098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.730945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.731098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.731018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.731098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.731063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkln\" (UniqueName: \"kubernetes.io/projected/de593aa0-7112-4d96-ba10-5725750c3072-kube-api-access-swkln\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.731708 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.731683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de593aa0-7112-4d96-ba10-5725750c3072-metrics-client-ca\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733452 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-grpc-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733593 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733571 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-tls\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733841 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733903 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.733903 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.733884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de593aa0-7112-4d96-ba10-5725750c3072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.738889 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.738869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkln\" (UniqueName: \"kubernetes.io/projected/de593aa0-7112-4d96-ba10-5725750c3072-kube-api-access-swkln\") pod \"thanos-querier-5d45c569c-wtk76\" (UID: \"de593aa0-7112-4d96-ba10-5725750c3072\") " pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:01:59.914002 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:01:59.913968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:02:00.066789 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:00.066757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d45c569c-wtk76"] Jul 15 08:02:00.072145 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:02:00.072097 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde593aa0_7112_4d96_ba10_5725750c3072.slice/crio-96a280474209b8b5b26ace4d3d9cc5b40db98bd0ce40efc4dd59bba28c262ec1 WatchSource:0}: Error finding container 96a280474209b8b5b26ace4d3d9cc5b40db98bd0ce40efc4dd59bba28c262ec1: Status 404 returned error can't find the container with id 96a280474209b8b5b26ace4d3d9cc5b40db98bd0ce40efc4dd59bba28c262ec1 Jul 15 08:02:00.557940 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:00.557902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"96a280474209b8b5b26ace4d3d9cc5b40db98bd0ce40efc4dd59bba28c262ec1"} Jul 15 08:02:00.559734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:00.559698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"cfc0bdff08386ebae42e5ccdf839ffef8b3c0b07b5f3dc17a21318eddbe9fad0"} Jul 15 08:02:01.424469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:01.424438 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ptzbf" Jul 15 08:02:01.566423 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:01.566374 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd" exitCode=0 Jul 15 08:02:01.566585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:01.566463 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd"} Jul 15 08:02:02.228035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.228004 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq"] Jul 15 08:02:02.238367 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.238278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.244901 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.244873 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 08:02:02.245180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.245162 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-47d2b\"" Jul 15 08:02:02.252644 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.252620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq"] Jul 15 08:02:02.357722 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.357683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxfs\" (UniqueName: \"kubernetes.io/projected/f19970a3-05fb-4dde-afb0-888b40392d21-kube-api-access-wxxfs\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.357899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.357745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f19970a3-05fb-4dde-afb0-888b40392d21-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.458813 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.458781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxfs\" (UniqueName: \"kubernetes.io/projected/f19970a3-05fb-4dde-afb0-888b40392d21-kube-api-access-wxxfs\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.459243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.458847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f19970a3-05fb-4dde-afb0-888b40392d21-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.461496 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.461472 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f19970a3-05fb-4dde-afb0-888b40392d21-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.470540 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.470516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxfs\" (UniqueName: \"kubernetes.io/projected/f19970a3-05fb-4dde-afb0-888b40392d21-kube-api-access-wxxfs\") pod \"monitoring-plugin-5f68df7f58-prvnq\" (UID: \"f19970a3-05fb-4dde-afb0-888b40392d21\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.548666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.548634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:02.659125 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.659088 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f4495d98-c2hmw"] Jul 15 08:02:02.664439 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.664387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.667221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667080 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 08:02:02.667221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667095 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 08:02:02.667221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667080 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 08:02:02.667221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667125 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 08:02:02.667546 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667481 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 08:02:02.667604 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.667549 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dqzzj\"" Jul 15 08:02:02.674216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.674186 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 08:02:02.680516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.680490 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f4495d98-c2hmw"] Jul 15 08:02:02.761961 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.761926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-serving-certs-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.761980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-metrics-client-ca\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk6f\" (UniqueName: \"kubernetes.io/projected/384ecd2f-e689-489e-b7dc-64d52dbfe41e-kube-api-access-khk6f\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.762452 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.762358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-federate-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.862957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.862873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk6f\" (UniqueName: \"kubernetes.io/projected/384ecd2f-e689-489e-b7dc-64d52dbfe41e-kube-api-access-khk6f\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.862957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.862932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.862963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.862986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-federate-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.863040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-serving-certs-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.863073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.863106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-metrics-client-ca\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863169 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.863140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.863999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.863947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-serving-certs-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.864163 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.864140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-metrics-client-ca\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.864396 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.864355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.866128 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.866101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.866238 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.866110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.866238 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.866215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-telemeter-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.866238 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.866228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/384ecd2f-e689-489e-b7dc-64d52dbfe41e-federate-client-tls\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.871474 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.871451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk6f\" (UniqueName: \"kubernetes.io/projected/384ecd2f-e689-489e-b7dc-64d52dbfe41e-kube-api-access-khk6f\") pod \"telemeter-client-7f4495d98-c2hmw\" (UID: \"384ecd2f-e689-489e-b7dc-64d52dbfe41e\") " pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:02.979703 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:02.979672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" Jul 15 08:02:03.639263 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.639226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq"] Jul 15 08:02:03.644064 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:02:03.644022 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19970a3_05fb_4dde_afb0_888b40392d21.slice/crio-378804283a8b92d1193678b6603c071ec9c70870d9f61bc4aaf4489aa169dd0e WatchSource:0}: Error finding container 378804283a8b92d1193678b6603c071ec9c70870d9f61bc4aaf4489aa169dd0e: Status 404 returned error can't find the container with id 378804283a8b92d1193678b6603c071ec9c70870d9f61bc4aaf4489aa169dd0e Jul 15 08:02:03.667692 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.667659 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:02:03.673305 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.673282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.679526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679254 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 08:02:03.679526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679500 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 08:02:03.679686 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679539 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 08:02:03.679743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 08:02:03.679743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679732 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679916 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.679949 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4dgml\"" Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f4495d98-c2hmw"] Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680319 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 08:02:03.680384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680346 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 08:02:03.680724 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680560 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 08:02:03.680815 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680796 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 08:02:03.680874 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.680812 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 08:02:03.681044 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.681026 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 08:02:03.681251 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.681231 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 08:02:03.681471 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.681453 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e45msojpfugbr\"" Jul 15 08:02:03.682653 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:02:03.682521 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384ecd2f_e689_489e_b7dc_64d52dbfe41e.slice/crio-b7550ee0b3798b0ea5719b45b428546ed3ce3c67cc351404886b586c194fb898 WatchSource:0}: Error finding container b7550ee0b3798b0ea5719b45b428546ed3ce3c67cc351404886b586c194fb898: Status 404 returned error can't find the container with id b7550ee0b3798b0ea5719b45b428546ed3ce3c67cc351404886b586c194fb898 Jul 15 08:02:03.683235 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.683147 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 08:02:03.691089 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.691067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:02:03.772339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.772615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcm4\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.773137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.772965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.873945 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.873903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.873956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.873986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.874526 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.874641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcm4\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875160 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.875061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875603 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.875392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.875813 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.875921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.875847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.876744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.876717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.878426 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.878315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.878610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.878527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.879317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.878930 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.879317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.879218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.879740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.879695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.879740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.879727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.880311 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.880287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.880567 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.880542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.881828 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.881802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.881929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.881815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.882808 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.882764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.883237 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.883211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.883352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.883333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.883688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.883667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.886109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.886087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcm4\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4\") pod \"prometheus-k8s-0\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:03.992880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:03.992841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:04.161221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.161188 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:02:04.162862 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:02:04.162835 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e6630f_e14e_478d_a732_d9c7a20023b5.slice/crio-8cc4be1df7d4c4b3c189da3cfd32966db27299a6fb238c30b078e42e160113c2 WatchSource:0}: Error finding container 8cc4be1df7d4c4b3c189da3cfd32966db27299a6fb238c30b078e42e160113c2: Status 404 returned error can't find the container with id 8cc4be1df7d4c4b3c189da3cfd32966db27299a6fb238c30b078e42e160113c2 Jul 15 08:02:04.435582 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.435501 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:02:04.581289 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.581243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" event={"ID":"f19970a3-05fb-4dde-afb0-888b40392d21","Type":"ContainerStarted","Data":"378804283a8b92d1193678b6603c071ec9c70870d9f61bc4aaf4489aa169dd0e"} Jul 15 08:02:04.582738 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.582710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"8cc4be1df7d4c4b3c189da3cfd32966db27299a6fb238c30b078e42e160113c2"} Jul 15 08:02:04.584438 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.584395 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b82e419-9f91-4efb-bf58-83caf297304a" containerID="f192d7581e61d83721174ba180de8c0106a70c568ad06a9ac8eb395410fef076" exitCode=0 Jul 15 08:02:04.584532 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.584507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6qsr" event={"ID":"6b82e419-9f91-4efb-bf58-83caf297304a","Type":"ContainerDied","Data":"f192d7581e61d83721174ba180de8c0106a70c568ad06a9ac8eb395410fef076"} Jul 15 08:02:04.585805 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.585780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" event={"ID":"384ecd2f-e689-489e-b7dc-64d52dbfe41e","Type":"ContainerStarted","Data":"b7550ee0b3798b0ea5719b45b428546ed3ce3c67cc351404886b586c194fb898"} Jul 15 08:02:04.588098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.588071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fcm7" event={"ID":"944e658e-f32c-449b-9be1-1f7d5e922c18","Type":"ContainerStarted","Data":"bf9f2790551dfb94cf1e9cb1618b77de82ca935c0502c84c1aaf722d5ef7fbd0"} Jul 15 08:02:04.625999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:04.625949 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5fcm7" podStartSLOduration=34.248572553 podStartE2EDuration="48.625933224s" totalImagesPullingTime="14.377360671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:01:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:04.624457066 +0000 UTC m=+102.044201873" watchObservedRunningTime="2026-07-15 08:02:04.625933224 +0000 UTC m=+102.045678026" Jul 15 08:02:05.594723 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.594639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"bb7e3b6816cafc8f18d4d9278c901ffd13aa6bf0252fce50164d4164a8188336"} Jul 15 08:02:05.595251 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.595227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"b455aad80c754982041acc51f71c3c55641ecd9c877107a3c8d2139a5e77812f"} Jul 15 08:02:05.595352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.595280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"2c1b832bcadba54bf23b7958840dd8c175dde1b0a142f971da99170a6105ce2b"} Jul 15 08:02:05.598216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.598188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648"} Jul 15 08:02:05.598323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.598244 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a"} Jul 15 08:02:05.598323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.598277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb"} Jul 15 08:02:05.598445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.598349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4"} Jul 15 08:02:05.598445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.598389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e"} Jul 15 08:02:05.599770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.599749 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" exitCode=0 Jul 15 08:02:05.599852 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.599830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} Jul 15 08:02:05.604452 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.604423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6qsr" event={"ID":"6b82e419-9f91-4efb-bf58-83caf297304a","Type":"ContainerStarted","Data":"6b75ca42b36fb81676d2030e9832c9c7d7cd1b38438bca7b425013eafe5c50ab"} Jul 15 08:02:05.604536 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.604480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6qsr" event={"ID":"6b82e419-9f91-4efb-bf58-83caf297304a","Type":"ContainerStarted","Data":"c335618429831f68ac8603026eff962b5d7d0c6ecc4de6323d9af3737d0b23a6"} Jul 15 08:02:05.653644 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:05.653587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c6qsr" podStartSLOduration=4.501546183 podStartE2EDuration="8.653571539s" totalImagesPullingTime="4.059798042s" totalInitContainerRuntime="92.227314ms" isStatefulPod=true podCreationTimestamp="2026-07-15 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:05.65188109 +0000 UTC m=+103.071625893" watchObservedRunningTime="2026-07-15 08:02:05.653571539 +0000 UTC m=+103.073316341" Jul 15 08:02:06.612859 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:06.612769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" event={"ID":"f19970a3-05fb-4dde-afb0-888b40392d21","Type":"ContainerStarted","Data":"1162204aeae9e2e419dc85d1101abdafce2714944e9f34d83e967bf43fa5ec06"} Jul 15 08:02:06.613649 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:06.613455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:06.619809 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:06.619759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" Jul 15 08:02:06.628905 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:06.628863 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-prvnq" podStartSLOduration=2.161668818 podStartE2EDuration="4.628848751s" totalImagesPullingTime="2.467179933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:06.628547788 +0000 UTC m=+104.048292604" watchObservedRunningTime="2026-07-15 08:02:06.628848751 +0000 UTC m=+104.048593553" Jul 15 08:02:07.622253 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.622055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerStarted","Data":"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74"} Jul 15 08:02:07.624397 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.624355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" event={"ID":"384ecd2f-e689-489e-b7dc-64d52dbfe41e","Type":"ContainerStarted","Data":"bb169837ab8840cc29b7f9f958d1ed48afbc34b6c8e5e62515e55acd5f29745a"} Jul 15 08:02:07.624529 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.624435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" event={"ID":"384ecd2f-e689-489e-b7dc-64d52dbfe41e","Type":"ContainerStarted","Data":"8938fba2d9cc4f2f20f489bf2dd2ae8ae389250f87e38a716f67684547bf6b9e"} Jul 15 08:02:07.624529 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.624462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" event={"ID":"384ecd2f-e689-489e-b7dc-64d52dbfe41e","Type":"ContainerStarted","Data":"c506e226ae900c33313e1cb1956cc34ecf8ef2030d741fe24c5472f74ee3148a"} Jul 15 08:02:07.627883 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.627845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"f3fe8a192a035872805c2a8874b336c507b9488d887e289631d758b7f6553f93"} Jul 15 08:02:07.628003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.627911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"8ed157a29a750dd82c6ebe1cdaf09bd30c3f33c9435f9356749b3a913550c573"} Jul 15 08:02:07.628003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.627941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" event={"ID":"de593aa0-7112-4d96-ba10-5725750c3072","Type":"ContainerStarted","Data":"ca8aa28e9681ac6d352b4a957e10886eccc52cfee37c0747a7d4616c65dfc484"} Jul 15 08:02:07.650805 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.650746 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.804702743 podStartE2EDuration="9.650725463s" totalImagesPullingTime="5.83074479s" totalInitContainerRuntime="15.27793ms" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:07.649193849 +0000 UTC m=+105.068938676" watchObservedRunningTime="2026-07-15 08:02:07.650725463 +0000 UTC m=+105.070470268" Jul 15 08:02:07.674564 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.674509 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" podStartSLOduration=2.565956485 podStartE2EDuration="8.674492708s" totalImagesPullingTime="6.108536223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:01:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:07.673199414 +0000 UTC m=+105.092944242" watchObservedRunningTime="2026-07-15 08:02:07.674492708 +0000 UTC m=+105.094237512" Jul 15 08:02:07.696644 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:07.696581 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f4495d98-c2hmw" podStartSLOduration=2.051463867 podStartE2EDuration="5.69655783s" totalImagesPullingTime="3.645093963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:07.694763622 +0000 UTC m=+105.114508482" watchObservedRunningTime="2026-07-15 08:02:07.69655783 +0000 UTC m=+105.116302636" Jul 15 08:02:08.632987 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:08.632954 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:02:09.644075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:09.643836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} Jul 15 08:02:09.644075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:09.643946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} Jul 15 08:02:09.644075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:09.643984 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} Jul 15 08:02:09.644075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:09.644024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} Jul 15 08:02:10.650720 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:10.650685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} Jul 15 08:02:10.651091 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:10.650743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerStarted","Data":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} Jul 15 08:02:10.681554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:10.681507 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.91740911 podStartE2EDuration="7.681490644s" totalImagesPullingTime="3.731074974s" totalInitContainerRuntime="33.00656ms" isStatefulPod=false podCreationTimestamp="2026-07-15 08:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:10.679505901 +0000 UTC m=+108.099250734" watchObservedRunningTime="2026-07-15 08:02:10.681490644 +0000 UTC m=+108.101235447" Jul 15 08:02:13.992960 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:13.992931 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:02:14.649924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:14.649897 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d45c569c-wtk76" Jul 15 08:02:18.921806 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:18.921770 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:02:26.721853 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:26.721820 2565 generic.go:356] "Generic (PLEG): container finished" podID="a0fe38cf-6ff3-4965-8bdf-2472b4abb40b" containerID="15900f624af5e0d547e4a0803cc3576544368ec3ffb08721651417c03c7a25b3" exitCode=0 Jul 15 08:02:26.722237 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:26.721899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" event={"ID":"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b","Type":"ContainerDied","Data":"15900f624af5e0d547e4a0803cc3576544368ec3ffb08721651417c03c7a25b3"} Jul 15 08:02:26.722283 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:26.722264 2565 scope.go:117] "RemoveContainer" containerID="15900f624af5e0d547e4a0803cc3576544368ec3ffb08721651417c03c7a25b3" Jul 15 08:02:27.727835 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:27.727801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sf7fl" event={"ID":"a0fe38cf-6ff3-4965-8bdf-2472b4abb40b","Type":"ContainerStarted","Data":"19160a32c8918598165e6cadbdcf58406026c469764236c076d2c261d1fd968f"} Jul 15 08:02:29.464509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:29.464465 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerName="registry" containerID="cri-o://4eb0a7726841155263e4f72e69dc118842fbdd21811dde42788a50f9c1e384d1" gracePeriod=30 Jul 15 08:02:29.739931 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:29.739899 2565 generic.go:356] "Generic (PLEG): container finished" podID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerID="4eb0a7726841155263e4f72e69dc118842fbdd21811dde42788a50f9c1e384d1" exitCode=0 Jul 15 08:02:29.740088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:29.739970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" event={"ID":"5dfe29f9-411a-4def-9aaf-41cbea161e06","Type":"ContainerDied","Data":"4eb0a7726841155263e4f72e69dc118842fbdd21811dde42788a50f9c1e384d1"} Jul 15 08:02:30.267726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.267702 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:02:30.334544 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334616 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334646 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxs76\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334881 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334714 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.334881 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.334741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.335209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.335175 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:30.335383 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.335288 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates\") pod \"5dfe29f9-411a-4def-9aaf-41cbea161e06\" (UID: \"5dfe29f9-411a-4def-9aaf-41cbea161e06\") " Jul 15 08:02:30.335773 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.335747 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:30.336095 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.336075 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-trusted-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.336191 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.336101 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-certificates\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.337445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.337414 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:02:30.337445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.337428 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:30.337701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.337673 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:02:30.337791 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.337692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:30.337791 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.337771 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76" (OuterVolumeSpecName: "kube-api-access-wxs76") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "kube-api-access-wxs76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:02:30.345606 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.345583 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5dfe29f9-411a-4def-9aaf-41cbea161e06" (UID: "5dfe29f9-411a-4def-9aaf-41cbea161e06"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437238 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-registry-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437268 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxs76\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-kube-api-access-wxs76\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437281 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-installation-pull-secrets\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437289 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5dfe29f9-411a-4def-9aaf-41cbea161e06-ca-trust-extracted\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437298 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5dfe29f9-411a-4def-9aaf-41cbea161e06-image-registry-private-configuration\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.437314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.437308 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dfe29f9-411a-4def-9aaf-41cbea161e06-bound-sa-token\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:30.745499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.745472 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" Jul 15 08:02:30.745499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.745491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66dcc4df8-htzcn" event={"ID":"5dfe29f9-411a-4def-9aaf-41cbea161e06","Type":"ContainerDied","Data":"967d8a688217034da23b375a7be06bc92ff6bc6c7631781e674e1bc376e76990"} Jul 15 08:02:30.745960 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.745545 2565 scope.go:117] "RemoveContainer" containerID="4eb0a7726841155263e4f72e69dc118842fbdd21811dde42788a50f9c1e384d1" Jul 15 08:02:30.767195 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.767172 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:02:30.770674 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:30.770650 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66dcc4df8-htzcn"] Jul 15 08:02:31.116415 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.116329 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" path="/var/lib/kubelet/pods/5dfe29f9-411a-4def-9aaf-41cbea161e06/volumes" Jul 15 08:02:31.750632 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.750600 2565 generic.go:356] "Generic (PLEG): container finished" podID="fba0308b-5cb6-4d38-bc52-f3d2aceb70cb" containerID="a98c252f0d2f2260813f19ee11e9bfba452061cec44c00356295a6962ca21b4a" exitCode=0 Jul 15 08:02:31.751032 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.750672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" event={"ID":"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb","Type":"ContainerDied","Data":"a98c252f0d2f2260813f19ee11e9bfba452061cec44c00356295a6962ca21b4a"} Jul 15 08:02:31.751032 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.751020 2565 scope.go:117] "RemoveContainer" containerID="a98c252f0d2f2260813f19ee11e9bfba452061cec44c00356295a6962ca21b4a" Jul 15 08:02:31.752190 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.752166 2565 generic.go:356] "Generic (PLEG): container finished" podID="f8d7f2ca-b85d-4084-9138-4d71c041be8a" containerID="fb1f62dedc581d1e390581790258fb29fb23d26e2544a47c19ee0d18f3ae5bfb" exitCode=0 Jul 15 08:02:31.752282 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.752234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" event={"ID":"f8d7f2ca-b85d-4084-9138-4d71c041be8a","Type":"ContainerDied","Data":"fb1f62dedc581d1e390581790258fb29fb23d26e2544a47c19ee0d18f3ae5bfb"} Jul 15 08:02:31.752503 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:31.752490 2565 scope.go:117] "RemoveContainer" containerID="fb1f62dedc581d1e390581790258fb29fb23d26e2544a47c19ee0d18f3ae5bfb" Jul 15 08:02:32.759369 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:32.759327 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mn54l" event={"ID":"fba0308b-5cb6-4d38-bc52-f3d2aceb70cb","Type":"ContainerStarted","Data":"f26732f0a743a4e02a6ddc65e2f92d4025ef2b52747381e866b3a70789a806b7"} Jul 15 08:02:32.761005 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:32.760985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-4gsng" event={"ID":"f8d7f2ca-b85d-4084-9138-4d71c041be8a","Type":"ContainerStarted","Data":"3ce812c8d15703f4e24463c6fa5b4632191994aa713665357fbc1eb655c36dd1"} Jul 15 08:02:32.960574 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:32.960524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:02:32.963071 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:32.963037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44438dfa-d58b-48f6-9002-0bbd27cf84c5-metrics-certs\") pod \"network-metrics-daemon-jh48p\" (UID: \"44438dfa-d58b-48f6-9002-0bbd27cf84c5\") " pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:02:33.125350 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:33.125273 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df2lv\"" Jul 15 08:02:33.133485 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:33.133462 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh48p" Jul 15 08:02:33.289296 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:33.289272 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jh48p"] Jul 15 08:02:33.291844 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:02:33.291816 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44438dfa_d58b_48f6_9002_0bbd27cf84c5.slice/crio-e1ebf90d072103086c59e2b9fcf16fa8d5673e879c066d2e852489bc135a38c0 WatchSource:0}: Error finding container e1ebf90d072103086c59e2b9fcf16fa8d5673e879c066d2e852489bc135a38c0: Status 404 returned error can't find the container with id e1ebf90d072103086c59e2b9fcf16fa8d5673e879c066d2e852489bc135a38c0 Jul 15 08:02:33.766643 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:33.766603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh48p" event={"ID":"44438dfa-d58b-48f6-9002-0bbd27cf84c5","Type":"ContainerStarted","Data":"e1ebf90d072103086c59e2b9fcf16fa8d5673e879c066d2e852489bc135a38c0"} Jul 15 08:02:35.777956 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:35.777913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh48p" event={"ID":"44438dfa-d58b-48f6-9002-0bbd27cf84c5","Type":"ContainerStarted","Data":"6321e5b0661ee1282901d889323dcddbe6312439200af6f02f9e8c3917b49f56"} Jul 15 08:02:35.778464 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:35.777969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh48p" event={"ID":"44438dfa-d58b-48f6-9002-0bbd27cf84c5","Type":"ContainerStarted","Data":"473b9f2a35ffb58a15e19c4a8b85dbb87ed2d77a517a22acde43e398be572ae9"} Jul 15 08:02:35.799835 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:35.799784 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jh48p" podStartSLOduration=131.739613979 podStartE2EDuration="2m13.799764975s" totalImagesPullingTime="2.060150996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:02:35.79871529 +0000 UTC m=+133.218460107" watchObservedRunningTime="2026-07-15 08:02:35.799764975 +0000 UTC m=+133.219509812" Jul 15 08:02:43.942821 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:43.942761 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55fcc7bb9-jnb6d" podUID="a90cf80b-b0ef-4539-ba23-14bd52451849" containerName="console" containerID="cri-o://169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2" gracePeriod=15 Jul 15 08:02:44.183320 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.183289 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:02:44.262395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262429 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262483 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262544 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrthz\" (UniqueName: \"kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262567 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca\") pod \"a90cf80b-b0ef-4539-ba23-14bd52451849\" (UID: \"a90cf80b-b0ef-4539-ba23-14bd52451849\") " Jul 15 08:02:44.262999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262955 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config" (OuterVolumeSpecName: "console-config") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:44.262999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.262973 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:44.263090 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.263001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca" (OuterVolumeSpecName: "service-ca") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:02:44.264730 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.264703 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz" (OuterVolumeSpecName: "kube-api-access-vrthz") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "kube-api-access-vrthz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:02:44.264836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.264784 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:44.264836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.264793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a90cf80b-b0ef-4539-ba23-14bd52451849" (UID: "a90cf80b-b0ef-4539-ba23-14bd52451849"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:02:44.363594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363544 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-oauth-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.363594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363587 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-console-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.363594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363598 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a90cf80b-b0ef-4539-ba23-14bd52451849-console-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.363594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363607 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-oauth-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.363853 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363616 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrthz\" (UniqueName: \"kubernetes.io/projected/a90cf80b-b0ef-4539-ba23-14bd52451849-kube-api-access-vrthz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.363853 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.363625 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a90cf80b-b0ef-4539-ba23-14bd52451849-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:02:44.819170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.819129 2565 generic.go:356] "Generic (PLEG): container finished" podID="a90cf80b-b0ef-4539-ba23-14bd52451849" containerID="169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2" exitCode=0 Jul 15 08:02:44.819170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.819171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fcc7bb9-jnb6d" event={"ID":"a90cf80b-b0ef-4539-ba23-14bd52451849","Type":"ContainerDied","Data":"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2"} Jul 15 08:02:44.819444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.819178 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fcc7bb9-jnb6d" Jul 15 08:02:44.819444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.819222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fcc7bb9-jnb6d" event={"ID":"a90cf80b-b0ef-4539-ba23-14bd52451849","Type":"ContainerDied","Data":"9e00676fb0f956ad86a4ee0227b3728bb3090e1a4f6bb9c836d54b5c0b799949"} Jul 15 08:02:44.819444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.819264 2565 scope.go:117] "RemoveContainer" containerID="169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2" Jul 15 08:02:44.827980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.827962 2565 scope.go:117] "RemoveContainer" containerID="169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2" Jul 15 08:02:44.828226 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:02:44.828207 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2\": container with ID starting with 169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2 not found: ID does not exist" containerID="169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2" Jul 15 08:02:44.828269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.828234 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2"} err="failed to get container status \"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2\": rpc error: code = NotFound desc = could not find container \"169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2\": container with ID starting with 169857ab0072aa2ee7eee0f08c16bd1cbf5c08a0397c3daf2b031c84900242f2 not found: ID does not exist" Jul 15 08:02:44.841397 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.841363 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:02:44.845880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:44.845854 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55fcc7bb9-jnb6d"] Jul 15 08:02:45.116230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:02:45.116157 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90cf80b-b0ef-4539-ba23-14bd52451849" path="/var/lib/kubelet/pods/a90cf80b-b0ef-4539-ba23-14bd52451849/volumes" Jul 15 08:03:03.993554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:03.993514 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:04.009494 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:04.009458 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:04.919788 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:04.919763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:17.991118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991036 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:17.991648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991332 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="alertmanager" containerID="cri-o://c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" gracePeriod=120 Jul 15 08:03:17.991648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991410 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-metric" containerID="cri-o://ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" gracePeriod=120 Jul 15 08:03:17.991648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991436 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy" containerID="cri-o://2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" gracePeriod=120 Jul 15 08:03:17.991648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991482 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="prom-label-proxy" containerID="cri-o://beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" gracePeriod=120 Jul 15 08:03:17.991921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991415 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-web" containerID="cri-o://36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" gracePeriod=120 Jul 15 08:03:17.991921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:17.991444 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="config-reloader" containerID="cri-o://106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" gracePeriod=120 Jul 15 08:03:18.967857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967822 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" exitCode=0 Jul 15 08:03:18.967857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967848 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" exitCode=0 Jul 15 08:03:18.967857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967856 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" exitCode=0 Jul 15 08:03:18.967857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967864 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" exitCode=0 Jul 15 08:03:18.968126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74"} Jul 15 08:03:18.968126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.967967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a"} Jul 15 08:03:18.968126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.968001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4"} Jul 15 08:03:18.968126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:18.968026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e"} Jul 15 08:03:19.234596 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.234573 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:19.279350 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279317 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279422 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279532 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279644 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkblb\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279713 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.279993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.279778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config\") pod \"62826120-d9a2-40b8-ba83-399605b51676\" (UID: \"62826120-d9a2-40b8-ba83-399605b51676\") " Jul 15 08:03:19.280141 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.280052 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:19.280348 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.280314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:03:19.284349 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.283999 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:19.284907 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.284862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.285062 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.285017 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.285423 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.285360 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:03:19.285527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.285453 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume" (OuterVolumeSpecName: "config-volume") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.286782 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.286743 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.287861 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.287827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb" (OuterVolumeSpecName: "kube-api-access-dkblb") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "kube-api-access-dkblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:03:19.288028 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.288005 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out" (OuterVolumeSpecName: "config-out") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:03:19.289184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.289146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.290995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.290919 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.298334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.298310 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config" (OuterVolumeSpecName: "web-config") pod "62826120-d9a2-40b8-ba83-399605b51676" (UID: "62826120-d9a2-40b8-ba83-399605b51676"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:19.380653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380610 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.380653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380644 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.380653 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380660 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-tls-assets\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380675 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-metrics-client-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380687 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-main-db\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380699 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62826120-d9a2-40b8-ba83-399605b51676-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380710 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380720 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkblb\" (UniqueName: \"kubernetes.io/projected/62826120-d9a2-40b8-ba83-399605b51676-kube-api-access-dkblb\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380731 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-cluster-tls-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380742 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380755 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-secret-alertmanager-main-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380766 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62826120-d9a2-40b8-ba83-399605b51676-web-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.381012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.380778 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62826120-d9a2-40b8-ba83-399605b51676-config-out\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:19.976198 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976162 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" exitCode=0 Jul 15 08:03:19.976198 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976193 2565 generic.go:356] "Generic (PLEG): container finished" podID="62826120-d9a2-40b8-ba83-399605b51676" containerID="36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" exitCode=0 Jul 15 08:03:19.976441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648"} Jul 15 08:03:19.976441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976248 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:19.976441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb"} Jul 15 08:03:19.976441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976333 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62826120-d9a2-40b8-ba83-399605b51676","Type":"ContainerDied","Data":"cfc0bdff08386ebae42e5ccdf839ffef8b3c0b07b5f3dc17a21318eddbe9fad0"} Jul 15 08:03:19.976441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.976373 2565 scope.go:117] "RemoveContainer" containerID="beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" Jul 15 08:03:19.986689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.986490 2565 scope.go:117] "RemoveContainer" containerID="ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" Jul 15 08:03:19.994049 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:19.994031 2565 scope.go:117] "RemoveContainer" containerID="2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" Jul 15 08:03:20.000849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.000833 2565 scope.go:117] "RemoveContainer" containerID="36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" Jul 15 08:03:20.007245 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.007221 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:20.008985 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.008963 2565 scope.go:117] "RemoveContainer" containerID="106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" Jul 15 08:03:20.012062 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.012040 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:20.017168 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.017152 2565 scope.go:117] "RemoveContainer" containerID="c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" Jul 15 08:03:20.024255 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.024240 2565 scope.go:117] "RemoveContainer" containerID="a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd" Jul 15 08:03:20.031165 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.031143 2565 scope.go:117] "RemoveContainer" containerID="beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" Jul 15 08:03:20.031445 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.031424 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74\": container with ID starting with beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74 not found: ID does not exist" containerID="beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" Jul 15 08:03:20.031509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.031454 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74"} err="failed to get container status \"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74\": rpc error: code = NotFound desc = could not find container \"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74\": container with ID starting with beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74 not found: ID does not exist" Jul 15 08:03:20.031509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.031474 2565 scope.go:117] "RemoveContainer" containerID="ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" Jul 15 08:03:20.031744 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.031716 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648\": container with ID starting with ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648 not found: ID does not exist" containerID="ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" Jul 15 08:03:20.031812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.031746 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648"} err="failed to get container status \"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648\": rpc error: code = NotFound desc = could not find container \"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648\": container with ID starting with ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648 not found: ID does not exist" Jul 15 08:03:20.031812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.031770 2565 scope.go:117] "RemoveContainer" containerID="2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" Jul 15 08:03:20.032024 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.032007 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a\": container with ID starting with 2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a not found: ID does not exist" containerID="2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" Jul 15 08:03:20.032060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032030 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a"} err="failed to get container status \"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a\": rpc error: code = NotFound desc = could not find container \"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a\": container with ID starting with 2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a not found: ID does not exist" Jul 15 08:03:20.032060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032044 2565 scope.go:117] "RemoveContainer" containerID="36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" Jul 15 08:03:20.032263 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.032245 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb\": container with ID starting with 36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb not found: ID does not exist" containerID="36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" Jul 15 08:03:20.032306 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032268 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb"} err="failed to get container status \"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb\": rpc error: code = NotFound desc = could not find container \"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb\": container with ID starting with 36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb not found: ID does not exist" Jul 15 08:03:20.032306 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032284 2565 scope.go:117] "RemoveContainer" containerID="106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" Jul 15 08:03:20.032496 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.032480 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4\": container with ID starting with 106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4 not found: ID does not exist" containerID="106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" Jul 15 08:03:20.032550 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032500 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4"} err="failed to get container status \"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4\": rpc error: code = NotFound desc = could not find container \"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4\": container with ID starting with 106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4 not found: ID does not exist" Jul 15 08:03:20.032550 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032515 2565 scope.go:117] "RemoveContainer" containerID="c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" Jul 15 08:03:20.032738 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.032722 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e\": container with ID starting with c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e not found: ID does not exist" containerID="c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" Jul 15 08:03:20.032798 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032745 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e"} err="failed to get container status \"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e\": rpc error: code = NotFound desc = could not find container \"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e\": container with ID starting with c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e not found: ID does not exist" Jul 15 08:03:20.032798 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.032764 2565 scope.go:117] "RemoveContainer" containerID="a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd" Jul 15 08:03:20.033002 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:20.032987 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd\": container with ID starting with a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd not found: ID does not exist" containerID="a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd" Jul 15 08:03:20.033045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033007 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd"} err="failed to get container status \"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd\": rpc error: code = NotFound desc = could not find container \"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd\": container with ID starting with a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd not found: ID does not exist" Jul 15 08:03:20.033045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033022 2565 scope.go:117] "RemoveContainer" containerID="beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74" Jul 15 08:03:20.033216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033199 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74"} err="failed to get container status \"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74\": rpc error: code = NotFound desc = could not find container \"beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74\": container with ID starting with beda78b780a46a9358938275228b7684457ed030d517996c8717fce088dc6b74 not found: ID does not exist" Jul 15 08:03:20.033256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033217 2565 scope.go:117] "RemoveContainer" containerID="ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648" Jul 15 08:03:20.033446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033429 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648"} err="failed to get container status \"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648\": rpc error: code = NotFound desc = could not find container \"ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648\": container with ID starting with ff54e0c2e70fbc692647d2a8f127c4c5ecb9deb033e17fe4f03d48605ae3b648 not found: ID does not exist" Jul 15 08:03:20.033518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033448 2565 scope.go:117] "RemoveContainer" containerID="2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a" Jul 15 08:03:20.033699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033682 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a"} err="failed to get container status \"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a\": rpc error: code = NotFound desc = could not find container \"2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a\": container with ID starting with 2f311b0a072ffed327566619c41cd9031cf179814e99dec9168bd86c9eef217a not found: ID does not exist" Jul 15 08:03:20.033744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033700 2565 scope.go:117] "RemoveContainer" containerID="36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb" Jul 15 08:03:20.033929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033905 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb"} err="failed to get container status \"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb\": rpc error: code = NotFound desc = could not find container \"36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb\": container with ID starting with 36566225ba752f4fa39d6b2d29b13f1f4c38948620eecef874dfc49c75ac5fcb not found: ID does not exist" Jul 15 08:03:20.033929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.033928 2565 scope.go:117] "RemoveContainer" containerID="106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4" Jul 15 08:03:20.034151 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.034133 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4"} err="failed to get container status \"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4\": rpc error: code = NotFound desc = could not find container \"106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4\": container with ID starting with 106394f630fa626a4df7a43275c293006a3806a81baf03419b305d4d410eceb4 not found: ID does not exist" Jul 15 08:03:20.034218 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.034152 2565 scope.go:117] "RemoveContainer" containerID="c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e" Jul 15 08:03:20.034344 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.034327 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e"} err="failed to get container status \"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e\": rpc error: code = NotFound desc = could not find container \"c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e\": container with ID starting with c028b13057e0ec99d1f5f25ad8c6db6581cec5b197276395ee3f62b75ad00b5e not found: ID does not exist" Jul 15 08:03:20.034414 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.034343 2565 scope.go:117] "RemoveContainer" containerID="a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd" Jul 15 08:03:20.034576 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.034559 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd"} err="failed to get container status \"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd\": rpc error: code = NotFound desc = could not find container \"a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd\": container with ID starting with a04bac6f1956ef3d5aad3390fd09ca26716326567bb28df2a5a8f532cb1b1dcd not found: ID does not exist" Jul 15 08:03:20.055388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055354 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:20.055722 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055707 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="config-reloader" Jul 15 08:03:20.055766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055739 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="config-reloader" Jul 15 08:03:20.055766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055752 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="prom-label-proxy" Jul 15 08:03:20.055766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055757 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="prom-label-proxy" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055771 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90cf80b-b0ef-4539-ba23-14bd52451849" containerName="console" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055777 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90cf80b-b0ef-4539-ba23-14bd52451849" containerName="console" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055784 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-web" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055790 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-web" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055798 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="init-config-reloader" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055804 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="init-config-reloader" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055810 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055816 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055825 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerName="registry" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055830 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerName="registry" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055837 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="alertmanager" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055842 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="alertmanager" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055847 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-metric" Jul 15 08:03:20.055857 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055852 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-metric" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055899 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe29f9-411a-4def-9aaf-41cbea161e06" containerName="registry" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055909 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90cf80b-b0ef-4539-ba23-14bd52451849" containerName="console" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055915 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="prom-label-proxy" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055922 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="alertmanager" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055927 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-web" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055933 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="config-reloader" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055939 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy-metric" Jul 15 08:03:20.056241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.055944 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62826120-d9a2-40b8-ba83-399605b51676" containerName="kube-rbac-proxy" Jul 15 08:03:20.061277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.061261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.064187 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064168 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 08:03:20.064345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 08:03:20.064472 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 08:03:20.064472 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064236 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 08:03:20.064472 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064238 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 08:03:20.064723 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064705 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 08:03:20.064778 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064708 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 08:03:20.064821 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064798 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 08:03:20.064821 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.064805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wfqqv\"" Jul 15 08:03:20.074715 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.074696 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 08:03:20.087280 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.087257 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:20.188234 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188234 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-web-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-config-out\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.188733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.188718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzr8r\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-kube-api-access-dzr8r\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289437 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-config-out\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzr8r\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-kube-api-access-dzr8r\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-web-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.289869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.289821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.291890 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.291175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.291890 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.291727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff2026aa-196e-4b26-9068-25385c82330f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.291890 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.291851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.292734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.292696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.292916 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.292891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff2026aa-196e-4b26-9068-25385c82330f-config-out\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293234 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-web-config\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.293818 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.293798 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.294034 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.294019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff2026aa-196e-4b26-9068-25385c82330f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.298889 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.298866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzr8r\" (UniqueName: \"kubernetes.io/projected/ff2026aa-196e-4b26-9068-25385c82330f-kube-api-access-dzr8r\") pod \"alertmanager-main-0\" (UID: \"ff2026aa-196e-4b26-9068-25385c82330f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.370574 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.370545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 08:03:20.512607 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.512570 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 08:03:20.981922 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.981887 2565 generic.go:356] "Generic (PLEG): container finished" podID="ff2026aa-196e-4b26-9068-25385c82330f" containerID="dd01e99ec90ab8cd16139876fb8b08138860242e427bcbcb3a5312f156bc5013" exitCode=0 Jul 15 08:03:20.982102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.981980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerDied","Data":"dd01e99ec90ab8cd16139876fb8b08138860242e427bcbcb3a5312f156bc5013"} Jul 15 08:03:20.982102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:20.982053 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"d4df0e27bad280ddb01d2a0993a4dbbc0bda6d387266847f525fbc171afa95f0"} Jul 15 08:03:21.115781 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.115754 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62826120-d9a2-40b8-ba83-399605b51676" path="/var/lib/kubelet/pods/62826120-d9a2-40b8-ba83-399605b51676/volumes" Jul 15 08:03:21.990730 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"825d6320d35d4529736f7c7c8d21c3af9432abae5d40a492fdaabcb157ec060e"} Jul 15 08:03:21.991097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"93a95317e51c17b82b630ec037600779935218d2268701a0b46c96004f0cd983"} Jul 15 08:03:21.991097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"dc2b059891283b7dda007a6e9d2bd1460043326d270ddac9f45e4fbbaaf6acef"} Jul 15 08:03:21.991097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"8a190ba1c26677c3b22c2ebd8b76347420398bc4e99c251818cf8e375f7ae91a"} Jul 15 08:03:21.991097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"ff070e844a7504c9689d9ea3fd17c751d29839195f8a372d5dd06934edd5c4e2"} Jul 15 08:03:21.991097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:21.990867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff2026aa-196e-4b26-9068-25385c82330f","Type":"ContainerStarted","Data":"5d6eccc1ed5868bca48ba9b6648a1834e7abd36b807e2d04298084ba93826cc7"} Jul 15 08:03:22.020258 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.020205 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.006332905 podStartE2EDuration="2.020189326s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.856421ms" isStatefulPod=false podCreationTimestamp="2026-07-15 08:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:03:22.018156123 +0000 UTC m=+179.437900954" watchObservedRunningTime="2026-07-15 08:03:22.020189326 +0000 UTC m=+179.439934129" Jul 15 08:03:22.256299 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256213 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:22.256548 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256508 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="prometheus" containerID="cri-o://981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" gracePeriod=600 Jul 15 08:03:22.256548 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256521 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy" containerID="cri-o://2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" gracePeriod=600 Jul 15 08:03:22.256742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256559 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" gracePeriod=600 Jul 15 08:03:22.256742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256531 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="thanos-sidecar" containerID="cri-o://9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" gracePeriod=600 Jul 15 08:03:22.256742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256540 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="config-reloader" containerID="cri-o://0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" gracePeriod=600 Jul 15 08:03:22.256742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.256571 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-web" containerID="cri-o://2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" gracePeriod=600 Jul 15 08:03:22.492870 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.492846 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:22.611248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611203 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611507 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611507 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611274 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611507 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611389 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611600 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611685 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611724 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611781 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611809 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611851 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.611911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611875 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611915 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611940 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611986 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.612011 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vcm4\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612143 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.612036 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1\") pod \"49e6630f-e14e-478d-a732-d9c7a20023b5\" (UID: \"49e6630f-e14e-478d-a732-d9c7a20023b5\") " Jul 15 08:03:22.612775 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.612775 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.611746 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.612775 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.612702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.613375 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.612724 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:03:22.613375 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.612957 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.613375 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.613253 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.614003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.613668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.614266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.614166 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.614266 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.614238 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.614914 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.614883 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out" (OuterVolumeSpecName: "config-out") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:03:22.615152 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.615116 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:03:22.615873 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.615805 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.615873 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.615854 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.616057 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.615900 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config" (OuterVolumeSpecName: "config") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.616057 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.615919 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:03:22.616235 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.616219 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.616373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.616357 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4" (OuterVolumeSpecName: "kube-api-access-7vcm4") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "kube-api-access-7vcm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:03:22.616744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.616724 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.616988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.616973 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.626331 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.626310 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config" (OuterVolumeSpecName: "web-config") pod "49e6630f-e14e-478d-a732-d9c7a20023b5" (UID: "49e6630f-e14e-478d-a732-d9c7a20023b5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713357 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713385 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-tls-assets\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713393 2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713417 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-metrics-client-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713426 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713437 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713445 2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-kube-rbac-proxy\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713453 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713464 2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-metrics-client-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713474 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713482 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcm4\" (UniqueName: \"kubernetes.io/projected/49e6630f-e14e-478d-a732-d9c7a20023b5-kube-api-access-7vcm4\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713490 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713498 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713508 2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713517 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713525 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-prometheus-k8s-db\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713533 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e6630f-e14e-478d-a732-d9c7a20023b5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713542 2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-secret-grpc-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713550 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49e6630f-e14e-478d-a732-d9c7a20023b5-config-out\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.713770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.713557 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49e6630f-e14e-478d-a732-d9c7a20023b5-web-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:03:22.998688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998653 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" exitCode=0 Jul 15 08:03:22.998688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998680 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" exitCode=0 Jul 15 08:03:22.998688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998686 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" exitCode=0 Jul 15 08:03:22.998688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998693 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" exitCode=0 Jul 15 08:03:22.998688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998698 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" exitCode=0 Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998704 2565 generic.go:356] "Generic (PLEG): container finished" podID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" exitCode=0 Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998728 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998875 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} Jul 15 08:03:22.999388 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:22.998985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49e6630f-e14e-478d-a732-d9c7a20023b5","Type":"ContainerDied","Data":"8cc4be1df7d4c4b3c189da3cfd32966db27299a6fb238c30b078e42e160113c2"} Jul 15 08:03:23.007205 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.007178 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.015462 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.015429 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.025077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.025060 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.034002 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.033984 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.041469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.041450 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.049301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.049265 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:23.053074 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.053051 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:23.061135 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.061114 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.069950 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.069926 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.070324 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.070289 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.070450 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.070338 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.070450 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.070364 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.070866 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.070846 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.070948 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.070870 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.070948 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.070885 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.071176 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.071155 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.071224 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071182 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.071224 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071197 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.071452 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.071433 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.071492 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071458 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.071492 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071472 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.071696 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.071680 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.071734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071700 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.071734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071712 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.071911 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.071890 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.071951 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071916 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.071951 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.071939 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.072175 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:03:23.072159 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.072216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072181 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.072216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072194 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.072393 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072376 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.072460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072394 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.072623 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072604 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.072670 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072624 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.072815 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072797 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.072868 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072815 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.073016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.072999 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.073076 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073018 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.073243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073225 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.073296 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073244 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.073509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073486 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.073509 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073509 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.073739 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073718 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.073785 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073743 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.073977 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073960 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.074022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.073977 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.074178 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074161 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.074225 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074179 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.074376 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074359 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.074446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074376 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.074620 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074602 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.074670 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074620 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.074824 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074808 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.074881 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.074826 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.075044 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075029 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.075102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075044 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.075239 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075220 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.075275 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075240 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.075477 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075459 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.075570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075479 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.075726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075704 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.075768 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075727 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.076004 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.075981 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.076065 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076000 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.076231 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076212 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.076302 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076232 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.076469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076442 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.076469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076461 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.076679 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076662 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.076727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076679 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.076880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076858 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.076880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.076881 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.077095 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077079 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.077136 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077095 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.077282 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077266 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.077330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077282 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.077487 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077470 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.077544 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077490 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.077716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077696 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.077755 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.077718 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.083137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083112 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.083242 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083138 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.083432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083391 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.083489 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083434 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.083694 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083677 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.083694 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083695 2565 scope.go:117] "RemoveContainer" containerID="337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8" Jul 15 08:03:23.083932 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083910 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8"} err="failed to get container status \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": rpc error: code = NotFound desc = could not find container \"337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8\": container with ID starting with 337b86e5a136e4d8ea900feb485a206eaf3ba7c53f58ace382d171fcee450aa8 not found: ID does not exist" Jul 15 08:03:23.083932 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.083930 2565 scope.go:117] "RemoveContainer" containerID="2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1" Jul 15 08:03:23.084182 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084162 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1"} err="failed to get container status \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": rpc error: code = NotFound desc = could not find container \"2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1\": container with ID starting with 2358043ed459a764cf05a52632059c90c3840827f5f3e258c11a52e8217882d1 not found: ID does not exist" Jul 15 08:03:23.084261 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084182 2565 scope.go:117] "RemoveContainer" containerID="2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a" Jul 15 08:03:23.084432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084393 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a"} err="failed to get container status \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": rpc error: code = NotFound desc = could not find container \"2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a\": container with ID starting with 2e11686bc99542fd80c1a17c4fbd20d2548700c560018ee62c59800285c0421a not found: ID does not exist" Jul 15 08:03:23.084480 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084433 2565 scope.go:117] "RemoveContainer" containerID="9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27" Jul 15 08:03:23.084691 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084669 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27"} err="failed to get container status \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": rpc error: code = NotFound desc = could not find container \"9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27\": container with ID starting with 9817bcf69ae59f3bb39342bf63c438cc94aeec716e4be6fae581a71352a92e27 not found: ID does not exist" Jul 15 08:03:23.084759 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084694 2565 scope.go:117] "RemoveContainer" containerID="0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c" Jul 15 08:03:23.084945 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084924 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c"} err="failed to get container status \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": rpc error: code = NotFound desc = could not find container \"0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c\": container with ID starting with 0eebdc95a670a380d34ea71b3b29d2589f310c6e31b11ae4e77616dc6e3bc82c not found: ID does not exist" Jul 15 08:03:23.085000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.084946 2565 scope.go:117] "RemoveContainer" containerID="981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95" Jul 15 08:03:23.085173 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.085155 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95"} err="failed to get container status \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": rpc error: code = NotFound desc = could not find container \"981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95\": container with ID starting with 981c9c1c098027560fa7edb4c83be7f75bcf20727f45562c34afeccb6824ae95 not found: ID does not exist" Jul 15 08:03:23.085229 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.085174 2565 scope.go:117] "RemoveContainer" containerID="340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360" Jul 15 08:03:23.085416 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.085376 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360"} err="failed to get container status \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": rpc error: code = NotFound desc = could not find container \"340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360\": container with ID starting with 340e20cb7680642df12cd6d4119549721dfc5def3a6a846c62e7acefdba2b360 not found: ID does not exist" Jul 15 08:03:23.090441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090422 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:23.090746 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090735 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy" Jul 15 08:03:23.090787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090748 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy" Jul 15 08:03:23.090787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090758 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-web" Jul 15 08:03:23.090787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090763 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-web" Jul 15 08:03:23.090787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090773 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="thanos-sidecar" Jul 15 08:03:23.090787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090778 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="thanos-sidecar" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090789 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="prometheus" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090795 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="prometheus" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090800 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="config-reloader" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090805 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="config-reloader" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090811 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-thanos" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090816 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-thanos" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090824 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="init-config-reloader" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090829 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="init-config-reloader" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090876 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090886 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="config-reloader" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090892 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="thanos-sidecar" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090897 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="prometheus" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090903 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-web" Jul 15 08:03:23.090924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.090908 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" containerName="kube-rbac-proxy-thanos" Jul 15 08:03:23.096204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.096190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.099088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099066 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4dgml\"" Jul 15 08:03:23.099207 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099192 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 08:03:23.099291 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099228 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 08:03:23.099515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099481 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 08:03:23.099515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099490 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 08:03:23.099657 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099554 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 08:03:23.099713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099672 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e45msojpfugbr\"" Jul 15 08:03:23.099765 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 08:03:23.099839 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099818 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 08:03:23.099986 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.099965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 08:03:23.100159 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.100147 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 08:03:23.100332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.100306 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 08:03:23.100440 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.100423 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 08:03:23.100537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.100515 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 08:03:23.101008 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.100992 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 08:03:23.102835 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.102785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 08:03:23.107665 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.107650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 08:03:23.116294 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.116269 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e6630f-e14e-478d-a732-d9c7a20023b5" path="/var/lib/kubelet/pods/49e6630f-e14e-478d-a732-d9c7a20023b5/volumes" Jul 15 08:03:23.118858 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.118837 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:23.217140 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217140 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-web-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-config-out\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp2x\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-kube-api-access-cvp2x\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.217979 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.217688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.318817 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-web-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.318817 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.318817 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-config-out\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.318817 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp2x\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-kube-api-access-cvp2x\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.318968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319118 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319394 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.319849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.319701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.320487 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.320461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.320749 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.320724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.320836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.320765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.321230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.321202 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.321890 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.321852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.322589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.322566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ce833d-f0db-4f19-946a-1880d5d0f09b-config-out\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.322770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.322745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-web-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.323530 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.323205 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.323530 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.323481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.324177 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.323895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.324177 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.324090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68ce833d-f0db-4f19-946a-1880d5d0f09b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.324177 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.324137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.324612 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.324590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.324750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.324728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.325116 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.325092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.325352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.325332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-config\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.325845 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.325824 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ce833d-f0db-4f19-946a-1880d5d0f09b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.328352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.328330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp2x\" (UniqueName: \"kubernetes.io/projected/68ce833d-f0db-4f19-946a-1880d5d0f09b-kube-api-access-cvp2x\") pod \"prometheus-k8s-0\" (UID: \"68ce833d-f0db-4f19-946a-1880d5d0f09b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.406313 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.406269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:03:23.556577 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:23.556445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 08:03:23.559113 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:03:23.559085 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ce833d_f0db_4f19_946a_1880d5d0f09b.slice/crio-e23840ff91fe6c8d495b54853f91290716987ef6f8ef5a298c5c31157f10cafa WatchSource:0}: Error finding container e23840ff91fe6c8d495b54853f91290716987ef6f8ef5a298c5c31157f10cafa: Status 404 returned error can't find the container with id e23840ff91fe6c8d495b54853f91290716987ef6f8ef5a298c5c31157f10cafa Jul 15 08:03:24.006532 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:24.006495 2565 generic.go:356] "Generic (PLEG): container finished" podID="68ce833d-f0db-4f19-946a-1880d5d0f09b" containerID="9b30256f1b2dbbc9c2e34d3db984bc8c17e7622f447204a95b3a46f15573be14" exitCode=0 Jul 15 08:03:24.006897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:24.006566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerDied","Data":"9b30256f1b2dbbc9c2e34d3db984bc8c17e7622f447204a95b3a46f15573be14"} Jul 15 08:03:24.006897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:24.006632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"e23840ff91fe6c8d495b54853f91290716987ef6f8ef5a298c5c31157f10cafa"} Jul 15 08:03:25.014551 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"7dfcdc545919e7081d459291bd5f3fa5216aee0565778bf0a39ed80e44eaea2c"} Jul 15 08:03:25.014958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"1c4203dce55a032b4410aabe6b4c76e27e967aca6c89e6efbd82485354c98731"} Jul 15 08:03:25.014958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"a5120e9045ef009b9354f8db18dd0f8daed47ff8cfd7f73739a8774b47397527"} Jul 15 08:03:25.014958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"77ebb8da438bae2ef3b835c4e1cbc1e0dc581209fb77fd9ff2cab83687d57fda"} Jul 15 08:03:25.014958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"af92045b7f2045d60eb4bb5986b181afb6bd298bc30eb0ba38f9bbeee4cd4938"} Jul 15 08:03:25.014958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.014698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68ce833d-f0db-4f19-946a-1880d5d0f09b","Type":"ContainerStarted","Data":"44044dcd1a32c76e2b1765d648c362aa27562139bbbfa3311fd6149380cfeef5"} Jul 15 08:03:25.069128 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:25.069076 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.050776552 podStartE2EDuration="2.06905909s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.282538ms" isStatefulPod=false podCreationTimestamp="2026-07-15 08:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:03:25.068567637 +0000 UTC m=+182.488312441" watchObservedRunningTime="2026-07-15 08:03:25.06905909 +0000 UTC m=+182.488803893" Jul 15 08:03:28.407170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:03:28.407133 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:04:23.407369 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:04:23.407271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:04:23.422796 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:04:23.422772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:04:24.269475 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:04:24.269445 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 08:05:21.412767 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.412733 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w"] Jul 15 08:05:21.416109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.416092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.418811 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.418790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 08:05:21.419500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.419478 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wtg45\"" Jul 15 08:05:21.419636 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.419581 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 08:05:21.419636 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.419570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 08:05:21.419768 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.419677 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 08:05:21.419768 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.419581 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 08:05:21.426446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.426423 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w"] Jul 15 08:05:21.472126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.472098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ktk\" (UniqueName: \"kubernetes.io/projected/08f26b44-40d4-4682-b10c-c6901fb418a6-kube-api-access-b2ktk\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.472269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.472148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08f26b44-40d4-4682-b10c-c6901fb418a6-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.472269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.472177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.472269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.472213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.573384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.573347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ktk\" (UniqueName: \"kubernetes.io/projected/08f26b44-40d4-4682-b10c-c6901fb418a6-kube-api-access-b2ktk\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.573610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.573433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08f26b44-40d4-4682-b10c-c6901fb418a6-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.573610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.573462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.573610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.573499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.574084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.574064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08f26b44-40d4-4682-b10c-c6901fb418a6-manager-config\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.575913 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.575891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-metrics-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.576011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.575896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f26b44-40d4-4682-b10c-c6901fb418a6-cert\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.586183 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.586159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ktk\" (UniqueName: \"kubernetes.io/projected/08f26b44-40d4-4682-b10c-c6901fb418a6-kube-api-access-b2ktk\") pod \"lws-controller-manager-5bdbd9fc98-mg24w\" (UID: \"08f26b44-40d4-4682-b10c-c6901fb418a6\") " pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.726026 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.725935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:21.861336 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:21.861223 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w"] Jul 15 08:05:21.863636 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:05:21.863610 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f26b44_40d4_4682_b10c_c6901fb418a6.slice/crio-d8cbb2bde6d40728cb954cbc411349656e9adfa1b25851a2d7deabb4059c45a3 WatchSource:0}: Error finding container d8cbb2bde6d40728cb954cbc411349656e9adfa1b25851a2d7deabb4059c45a3: Status 404 returned error can't find the container with id d8cbb2bde6d40728cb954cbc411349656e9adfa1b25851a2d7deabb4059c45a3 Jul 15 08:05:22.502911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:22.502870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" event={"ID":"08f26b44-40d4-4682-b10c-c6901fb418a6","Type":"ContainerStarted","Data":"d8cbb2bde6d40728cb954cbc411349656e9adfa1b25851a2d7deabb4059c45a3"} Jul 15 08:05:23.026633 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:23.026596 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:05:23.036444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:23.036397 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:05:23.039610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:23.039587 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:05:23.045434 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:23.045389 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 08:05:23.046929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:23.046841 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:05:25.517588 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:25.517551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" event={"ID":"08f26b44-40d4-4682-b10c-c6901fb418a6","Type":"ContainerStarted","Data":"e5cf6e9b91dfa823668b523b931e675a8b53aef8bdc8db80e491bb2e93b1a93b"} Jul 15 08:05:25.517993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:25.517688 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:25.539022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:25.538976 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" podStartSLOduration=1.903962547 podStartE2EDuration="4.538962287s" totalImagesPullingTime="2.63499974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:05:25.537826723 +0000 UTC m=+302.957571526" watchObservedRunningTime="2026-07-15 08:05:25.538962287 +0000 UTC m=+302.958707137" Jul 15 08:05:31.495542 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.495510 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn"] Jul 15 08:05:31.498965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.498951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.501310 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.501287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 08:05:31.501451 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.501354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 08:05:31.501861 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.501835 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 08:05:31.501965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.501865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5958h\"" Jul 15 08:05:31.502022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.501979 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 08:05:31.519440 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.519415 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn"] Jul 15 08:05:31.565524 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.565489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.565726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.565554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.565726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.565633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.565726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.565656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh842\" (UniqueName: \"kubernetes.io/projected/d49efb9a-8e4a-41a2-84e3-66e3b5329698-kube-api-access-sh842\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.565726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.565683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.666815 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.666780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.667017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.666829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.667017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.666891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.667017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.666909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh842\" (UniqueName: \"kubernetes.io/projected/d49efb9a-8e4a-41a2-84e3-66e3b5329698-kube-api-access-sh842\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.667017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.666933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.667337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.667315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-manifests\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.669201 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.669175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49efb9a-8e4a-41a2-84e3-66e3b5329698-tmp\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.669455 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.669438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.669555 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.669536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49efb9a-8e4a-41a2-84e3-66e3b5329698-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.678896 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.678874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh842\" (UniqueName: \"kubernetes.io/projected/d49efb9a-8e4a-41a2-84e3-66e3b5329698-kube-api-access-sh842\") pod \"opendatahub-operator-controller-manager-7b8c85cff6-2g5cn\" (UID: \"d49efb9a-8e4a-41a2-84e3-66e3b5329698\") " pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.813661 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.813576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:31.984060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.983979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn"] Jul 15 08:05:31.986852 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:05:31.986827 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49efb9a_8e4a_41a2_84e3_66e3b5329698.slice/crio-000c178ec6cd5386332f28005641ac532f6b1eb3af7e4ecdeb1c6c6ece3ab182 WatchSource:0}: Error finding container 000c178ec6cd5386332f28005641ac532f6b1eb3af7e4ecdeb1c6c6ece3ab182: Status 404 returned error can't find the container with id 000c178ec6cd5386332f28005641ac532f6b1eb3af7e4ecdeb1c6c6ece3ab182 Jul 15 08:05:31.988634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:31.988614 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:05:32.548811 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:32.548775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" event={"ID":"d49efb9a-8e4a-41a2-84e3-66e3b5329698","Type":"ContainerStarted","Data":"000c178ec6cd5386332f28005641ac532f6b1eb3af7e4ecdeb1c6c6ece3ab182"} Jul 15 08:05:35.564706 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:35.564616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" event={"ID":"d49efb9a-8e4a-41a2-84e3-66e3b5329698","Type":"ContainerStarted","Data":"ec7579d2e7e22ff591484b35a42a08cb40e9516356b69d1c0b2c89fccc0baa7e"} Jul 15 08:05:36.525216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:36.525184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bdbd9fc98-mg24w" Jul 15 08:05:36.570717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:36.570684 2565 generic.go:356] "Generic (PLEG): container finished" podID="d49efb9a-8e4a-41a2-84e3-66e3b5329698" containerID="ec7579d2e7e22ff591484b35a42a08cb40e9516356b69d1c0b2c89fccc0baa7e" exitCode=0 Jul 15 08:05:36.571140 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:36.570768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" event={"ID":"d49efb9a-8e4a-41a2-84e3-66e3b5329698","Type":"ContainerDied","Data":"ec7579d2e7e22ff591484b35a42a08cb40e9516356b69d1c0b2c89fccc0baa7e"} Jul 15 08:05:37.576698 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:37.576662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" event={"ID":"d49efb9a-8e4a-41a2-84e3-66e3b5329698","Type":"ContainerStarted","Data":"b2edddc74933bff26fa34a429e08539c12774fd7504efc6208467d031bcd8f51"} Jul 15 08:05:37.577084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:37.576762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:37.609055 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:37.609007 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" podStartSLOduration=1.8590249970000001 podStartE2EDuration="6.608993566s" totalImagesPullingTime="3.691863812s" totalInitContainerRuntime="1.058104757s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:05:37.607690081 +0000 UTC m=+315.027434903" watchObservedRunningTime="2026-07-15 08:05:37.608993566 +0000 UTC m=+315.028738369" Jul 15 08:05:48.583361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:48.583332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b8c85cff6-2g5cn" Jul 15 08:05:58.889247 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.889210 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622"] Jul 15 08:05:58.899493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.899473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:58.901781 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.901756 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 08:05:58.901910 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.901785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 08:05:58.901910 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.901799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b6j64\"" Jul 15 08:05:58.902173 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.902159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 08:05:58.905052 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.905033 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 08:05:58.910300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:58.910281 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622"] Jul 15 08:05:59.012122 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012122 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b062e0e0-1066-43a8-853b-9de1b131d4d5-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxvq\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-kube-api-access-6wxvq\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.012364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.012349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.112850 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.112806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.112878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.112898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.112921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b062e0e0-1066-43a8-853b-9de1b131d4d5-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.112974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113242 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.113028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113242 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.113053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxvq\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-kube-api-access-6wxvq\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.113845 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.113812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.115492 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.115457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.115608 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.115502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.115717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.115684 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b062e0e0-1066-43a8-853b-9de1b131d4d5-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.115755 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.115742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.132212 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.132166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.134939 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.134917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxvq\" (UniqueName: \"kubernetes.io/projected/b062e0e0-1066-43a8-853b-9de1b131d4d5-kube-api-access-6wxvq\") pod \"istiod-openshift-gateway-94bb8fbfd-m4622\" (UID: \"b062e0e0-1066-43a8-853b-9de1b131d4d5\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.209003 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.208924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:05:59.348343 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.348311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622"] Jul 15 08:05:59.351375 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:05:59.351350 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb062e0e0_1066_43a8_853b_9de1b131d4d5.slice/crio-1885ad575e056ba2602060211455fb0d2c40bb503c0435eff37a15adb9c0cd91 WatchSource:0}: Error finding container 1885ad575e056ba2602060211455fb0d2c40bb503c0435eff37a15adb9c0cd91: Status 404 returned error can't find the container with id 1885ad575e056ba2602060211455fb0d2c40bb503c0435eff37a15adb9c0cd91 Jul 15 08:05:59.674598 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:05:59.674558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" event={"ID":"b062e0e0-1066-43a8-853b-9de1b131d4d5","Type":"ContainerStarted","Data":"1885ad575e056ba2602060211455fb0d2c40bb503c0435eff37a15adb9c0cd91"} Jul 15 08:06:01.791950 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:01.791914 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:06:01.792196 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:01.791982 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:06:02.692049 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:02.692016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" event={"ID":"b062e0e0-1066-43a8-853b-9de1b131d4d5","Type":"ContainerStarted","Data":"1b8d3082c67f7dcdcfca62d2f3d9366a76660064e4487672eceb0db215398368"} Jul 15 08:06:02.692248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:02.692230 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:06:02.694037 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:02.694003 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" Jul 15 08:06:02.719223 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:06:02.719164 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m4622" podStartSLOduration=2.280785395 podStartE2EDuration="4.719147531s" totalImagesPullingTime="2.438362136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:06:02.717963944 +0000 UTC m=+340.137708748" watchObservedRunningTime="2026-07-15 08:06:02.719147531 +0000 UTC m=+340.138892335" Jul 15 08:07:53.749839 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.745068 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p"] Jul 15 08:07:53.750353 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.750332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:53.752996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.752968 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 08:07:53.753119 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.753001 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 08:07:53.753119 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.753055 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fgfbk\"" Jul 15 08:07:53.768286 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.768261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p"] Jul 15 08:07:53.901171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.901140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8ht\" (UniqueName: \"kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:53.901324 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:53.901185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.002433 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.002327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8ht\" (UniqueName: \"kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.002433 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.002393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.002727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.002706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.013161 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.013138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8ht\" (UniqueName: \"kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht\") pod \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.014757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.014738 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p"] Jul 15 08:07:54.014979 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.014968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:54.020800 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.020778 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p"] Jul 15 08:07:54.056219 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.056191 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:07:54.059633 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.059618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.077333 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.077305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:07:54.204959 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.204926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.205117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.204995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqz75\" (UniqueName: \"kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.306299 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.306216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.306561 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.306534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqz75\" (UniqueName: \"kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.306939 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.306913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.320884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.320858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqz75\" (UniqueName: \"kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75\") pod \"kuadrant-operator-controller-manager-55467d9f7-dzsh9\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.370371 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.370348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:54.517836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:54.517735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:07:54.520743 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:07:54.520720 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee95d4b3_65f8_4d2c_a6cc_20173cc009f0.slice/crio-f4e6edc01ef89818942543531e6a9cdcf3dde1db6f02a1118a37bd3dcd66c3db WatchSource:0}: Error finding container f4e6edc01ef89818942543531e6a9cdcf3dde1db6f02a1118a37bd3dcd66c3db: Status 404 returned error can't find the container with id f4e6edc01ef89818942543531e6a9cdcf3dde1db6f02a1118a37bd3dcd66c3db Jul 15 08:07:55.206339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:55.206306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" event={"ID":"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0","Type":"ContainerStarted","Data":"f4e6edc01ef89818942543531e6a9cdcf3dde1db6f02a1118a37bd3dcd66c3db"} Jul 15 08:07:56.569062 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:07:56.569029 2565 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-mhw7p_kuadrant-system_a95faf32-afb3-48e3-a14c-5a7f09370bcb_0(098d63e89587b4c5eb71feddd78ac159765fb651128a2180f286f397b5ea010b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-mhw7p to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 15 08:07:56.569624 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:07:56.569094 2565 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-mhw7p_kuadrant-system_a95faf32-afb3-48e3-a14c-5a7f09370bcb_0(098d63e89587b4c5eb71feddd78ac159765fb651128a2180f286f397b5ea010b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-mhw7p to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:57.216454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.216422 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:57.225413 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.225372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:57.229155 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:07:57.229123 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="a95faf32-afb3-48e3-a14c-5a7f09370bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:57.333979 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.333951 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj8ht\" (UniqueName: \"kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht\") pod \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " Jul 15 08:07:57.334137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.334061 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume\") pod \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\" (UID: \"a95faf32-afb3-48e3-a14c-5a7f09370bcb\") " Jul 15 08:07:57.334333 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.334303 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a95faf32-afb3-48e3-a14c-5a7f09370bcb" (UID: "a95faf32-afb3-48e3-a14c-5a7f09370bcb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:07:57.336099 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.336075 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht" (OuterVolumeSpecName: "kube-api-access-cj8ht") pod "a95faf32-afb3-48e3-a14c-5a7f09370bcb" (UID: "a95faf32-afb3-48e3-a14c-5a7f09370bcb"). InnerVolumeSpecName "kube-api-access-cj8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:07:57.435587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.435557 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a95faf32-afb3-48e3-a14c-5a7f09370bcb-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:07:57.435587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:57.435584 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8ht\" (UniqueName: \"kubernetes.io/projected/a95faf32-afb3-48e3-a14c-5a7f09370bcb-kube-api-access-cj8ht\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:07:58.221068 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:58.221032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:58.226859 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:07:58.226735 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="a95faf32-afb3-48e3-a14c-5a7f09370bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:58.231667 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:07:58.231585 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-mhw7p\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="a95faf32-afb3-48e3-a14c-5a7f09370bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-mhw7p" Jul 15 08:07:59.116808 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:59.116780 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a95faf32-afb3-48e3-a14c-5a7f09370bcb" path="/var/lib/kubelet/pods/a95faf32-afb3-48e3-a14c-5a7f09370bcb/volumes" Jul 15 08:07:59.229387 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:59.229346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" event={"ID":"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0","Type":"ContainerStarted","Data":"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938"} Jul 15 08:07:59.230146 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:59.230118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:07:59.257376 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:07:59.257311 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" podStartSLOduration=0.680387218 podStartE2EDuration="5.257291109s" totalImagesPullingTime="4.576903891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:07:59.253059156 +0000 UTC m=+456.672803957" watchObservedRunningTime="2026-07-15 08:07:59.257291109 +0000 UTC m=+456.677035911" Jul 15 08:08:11.242046 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:11.242017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:08:23.371734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.371697 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:08:23.372160 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.371919 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" podUID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" containerName="manager" containerID="cri-o://0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938" gracePeriod=10 Jul 15 08:08:23.624923 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.624868 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:08:23.656532 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.656506 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume\") pod \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " Jul 15 08:08:23.656626 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.656557 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqz75\" (UniqueName: \"kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75\") pod \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\" (UID: \"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0\") " Jul 15 08:08:23.656900 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.656866 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" (UID: "ee95d4b3-65f8-4d2c-a6cc-20173cc009f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:08:23.658498 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.658477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75" (OuterVolumeSpecName: "kube-api-access-qqz75") pod "ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" (UID: "ee95d4b3-65f8-4d2c-a6cc-20173cc009f0"). InnerVolumeSpecName "kube-api-access-qqz75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:08:23.757539 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.757515 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqz75\" (UniqueName: \"kubernetes.io/projected/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-kube-api-access-qqz75\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:08:23.757539 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:23.757537 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:08:24.347126 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.347092 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" containerID="0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938" exitCode=0 Jul 15 08:08:24.347334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.347158 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" Jul 15 08:08:24.347334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.347168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" event={"ID":"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0","Type":"ContainerDied","Data":"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938"} Jul 15 08:08:24.347334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.347232 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9" event={"ID":"ee95d4b3-65f8-4d2c-a6cc-20173cc009f0","Type":"ContainerDied","Data":"f4e6edc01ef89818942543531e6a9cdcf3dde1db6f02a1118a37bd3dcd66c3db"} Jul 15 08:08:24.347334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.347252 2565 scope.go:117] "RemoveContainer" containerID="0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938" Jul 15 08:08:24.356851 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.356828 2565 scope.go:117] "RemoveContainer" containerID="0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938" Jul 15 08:08:24.357120 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:08:24.357100 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938\": container with ID starting with 0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938 not found: ID does not exist" containerID="0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938" Jul 15 08:08:24.357182 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.357128 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938"} err="failed to get container status \"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938\": rpc error: code = NotFound desc = could not find container \"0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938\": container with ID starting with 0e2c0c26589628bfecce6a43bdd3688d653079fab188e451cf137bf4077a5938 not found: ID does not exist" Jul 15 08:08:24.375296 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.375271 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:08:24.380479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:24.380454 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dzsh9"] Jul 15 08:08:25.116021 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:25.115989 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" path="/var/lib/kubelet/pods/ee95d4b3-65f8-4d2c-a6cc-20173cc009f0/volumes" Jul 15 08:08:42.914883 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.914849 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:08:42.915324 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.915190 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" containerName="manager" Jul 15 08:08:42.915324 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.915200 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" containerName="manager" Jul 15 08:08:42.915324 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.915268 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee95d4b3-65f8-4d2c-a6cc-20173cc009f0" containerName="manager" Jul 15 08:08:42.919876 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.919856 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:42.922101 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.922080 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 08:08:42.922278 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.922263 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 08:08:42.922859 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.922841 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 08:08:42.922922 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.922841 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rx7jh\"" Jul 15 08:08:42.927752 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:42.927733 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:08:43.016065 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.016028 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:08:43.022317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.022289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.022508 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.022332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwhk\" (UniqueName: \"kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.123392 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.123363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.123564 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.123412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwhk\" (UniqueName: \"kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.123983 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.123963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.131764 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.131744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwhk\" (UniqueName: \"kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk\") pod \"limitador-limitador-64c94fd64f-j46zq\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.230259 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.230235 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:43.356209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.356181 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:08:43.358017 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:08:43.357989 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31010cb4_c96b_4650_89b5_dd5ee921aa28.slice/crio-cfb28961d99fe39104e0f744708055a599c7727400d78dd1d4b924972a944b5c WatchSource:0}: Error finding container cfb28961d99fe39104e0f744708055a599c7727400d78dd1d4b924972a944b5c: Status 404 returned error can't find the container with id cfb28961d99fe39104e0f744708055a599c7727400d78dd1d4b924972a944b5c Jul 15 08:08:43.436294 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:43.436262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" event={"ID":"31010cb4-c96b-4650-89b5-dd5ee921aa28","Type":"ContainerStarted","Data":"cfb28961d99fe39104e0f744708055a599c7727400d78dd1d4b924972a944b5c"} Jul 15 08:08:46.454738 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:46.454653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" event={"ID":"31010cb4-c96b-4650-89b5-dd5ee921aa28","Type":"ContainerStarted","Data":"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799"} Jul 15 08:08:46.454738 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:46.454720 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:46.476287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:46.476244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" podStartSLOduration=1.811029572 podStartE2EDuration="4.476230557s" totalImagesPullingTime="2.665200985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:08:46.474553614 +0000 UTC m=+503.894298418" watchObservedRunningTime="2026-07-15 08:08:46.476230557 +0000 UTC m=+503.895975358" Jul 15 08:08:57.459786 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:57.459755 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:59.352839 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.352801 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:08:59.353300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.352964 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" podUID="31010cb4-c96b-4650-89b5-dd5ee921aa28" containerName="limitador" containerID="cri-o://d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799" gracePeriod=30 Jul 15 08:08:59.899170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.899148 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:08:59.972793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.972715 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file\") pod \"31010cb4-c96b-4650-89b5-dd5ee921aa28\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " Jul 15 08:08:59.972793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.972770 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vwhk\" (UniqueName: \"kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk\") pod \"31010cb4-c96b-4650-89b5-dd5ee921aa28\" (UID: \"31010cb4-c96b-4650-89b5-dd5ee921aa28\") " Jul 15 08:08:59.973057 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.973033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file" (OuterVolumeSpecName: "config-file") pod "31010cb4-c96b-4650-89b5-dd5ee921aa28" (UID: "31010cb4-c96b-4650-89b5-dd5ee921aa28"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:08:59.974756 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:08:59.974739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk" (OuterVolumeSpecName: "kube-api-access-6vwhk") pod "31010cb4-c96b-4650-89b5-dd5ee921aa28" (UID: "31010cb4-c96b-4650-89b5-dd5ee921aa28"). InnerVolumeSpecName "kube-api-access-6vwhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:09:00.073630 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.073598 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwhk\" (UniqueName: \"kubernetes.io/projected/31010cb4-c96b-4650-89b5-dd5ee921aa28-kube-api-access-6vwhk\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:09:00.073630 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.073621 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31010cb4-c96b-4650-89b5-dd5ee921aa28-config-file\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:09:00.519267 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.519234 2565 generic.go:356] "Generic (PLEG): container finished" podID="31010cb4-c96b-4650-89b5-dd5ee921aa28" containerID="d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799" exitCode=0 Jul 15 08:09:00.519714 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.519275 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" Jul 15 08:09:00.519714 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.519307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" event={"ID":"31010cb4-c96b-4650-89b5-dd5ee921aa28","Type":"ContainerDied","Data":"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799"} Jul 15 08:09:00.519714 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.519344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j46zq" event={"ID":"31010cb4-c96b-4650-89b5-dd5ee921aa28","Type":"ContainerDied","Data":"cfb28961d99fe39104e0f744708055a599c7727400d78dd1d4b924972a944b5c"} Jul 15 08:09:00.519714 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.519366 2565 scope.go:117] "RemoveContainer" containerID="d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799" Jul 15 08:09:00.529819 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.529800 2565 scope.go:117] "RemoveContainer" containerID="d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799" Jul 15 08:09:00.530071 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:09:00.530052 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799\": container with ID starting with d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799 not found: ID does not exist" containerID="d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799" Jul 15 08:09:00.530121 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.530078 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799"} err="failed to get container status \"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799\": rpc error: code = NotFound desc = could not find container \"d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799\": container with ID starting with d44bb48f96dfd7ec3b956002c818532ff5e203f2748c532e7c2ad28f4e3ce799 not found: ID does not exist" Jul 15 08:09:00.554130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.554105 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:09:00.564734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:00.564714 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j46zq"] Jul 15 08:09:01.116556 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.116525 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31010cb4-c96b-4650-89b5-dd5ee921aa28" path="/var/lib/kubelet/pods/31010cb4-c96b-4650-89b5-dd5ee921aa28/volumes" Jul 15 08:09:01.750830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.750798 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb"] Jul 15 08:09:01.751196 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.751173 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31010cb4-c96b-4650-89b5-dd5ee921aa28" containerName="limitador" Jul 15 08:09:01.751196 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.751184 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31010cb4-c96b-4650-89b5-dd5ee921aa28" containerName="limitador" Jul 15 08:09:01.751269 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.751258 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31010cb4-c96b-4650-89b5-dd5ee921aa28" containerName="limitador" Jul 15 08:09:01.755719 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.755703 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.764116 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.764099 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 08:09:01.764355 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.764343 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 08:09:01.764568 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.764556 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-l7hgx\"" Jul 15 08:09:01.765736 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.765722 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 08:09:01.787933 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.787900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-data\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.788055 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.787934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtjm\" (UniqueName: \"kubernetes.io/projected/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-kube-api-access-wrtjm\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.791183 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.791159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb"] Jul 15 08:09:01.888757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.888725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-data\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.888757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.888758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtjm\" (UniqueName: \"kubernetes.io/projected/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-kube-api-access-wrtjm\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.889153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.889132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-data\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:01.919202 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:01.919179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtjm\" (UniqueName: \"kubernetes.io/projected/a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe-kube-api-access-wrtjm\") pod \"postgres-69fb4594cf-vn5zb\" (UID: \"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:02.065630 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:02.065540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:02.211023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:02.210999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb"] Jul 15 08:09:02.212909 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:02.212877 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f76b51_84dd_4c76_b3b4_a1461e5ce7fe.slice/crio-d231f7dc01869ec2eaaf4402969eec7504c3f53d891ed8094a5203f1708f5dfb WatchSource:0}: Error finding container d231f7dc01869ec2eaaf4402969eec7504c3f53d891ed8094a5203f1708f5dfb: Status 404 returned error can't find the container with id d231f7dc01869ec2eaaf4402969eec7504c3f53d891ed8094a5203f1708f5dfb Jul 15 08:09:02.531276 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:02.531242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" event={"ID":"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe","Type":"ContainerStarted","Data":"d231f7dc01869ec2eaaf4402969eec7504c3f53d891ed8094a5203f1708f5dfb"} Jul 15 08:09:07.560503 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:07.560458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" event={"ID":"a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe","Type":"ContainerStarted","Data":"ef5ee34bd26301097ad81b062eb42868e8cf651a3e18b011e9b2eb70b6edb05b"} Jul 15 08:09:07.560950 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:07.560538 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:07.582051 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:07.581953 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" podStartSLOduration=1.470326418 podStartE2EDuration="6.581937142s" totalImagesPullingTime="5.111610724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:09:07.58115923 +0000 UTC m=+525.000904058" watchObservedRunningTime="2026-07-15 08:09:07.581937142 +0000 UTC m=+525.001681944" Jul 15 08:09:13.594568 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:13.594540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vn5zb" Jul 15 08:09:21.544988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.544956 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:21.553930 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.553901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.556584 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.556563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 08:09:21.557047 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.557032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6wnqs\"" Jul 15 08:09:21.557161 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.557080 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 08:09:21.559669 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.559647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:21.665025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.664993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.665180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.665028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.732151 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.732120 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:09:21.735537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.735519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.744650 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.744620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:09:21.766216 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.766188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.766336 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.766224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.768517 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.768500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.775140 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.775120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj\") pod \"maas-controller-7dd86fd965-825js\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.865889 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.865812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:21.867175 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.867151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.867277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.867196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7p5\" (UniqueName: \"kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.968022 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.967988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7p5\" (UniqueName: \"kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.968189 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.968106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.970864 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.970833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:21.976936 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:21.976911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7p5\" (UniqueName: \"kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5\") pod \"maas-controller-599b49958f-xjq7b\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:22.045689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:22.045661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:22.170461 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:22.170440 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:09:22.172268 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:22.172240 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b388cfc_9218_4bc8_931e_c2653aa9d464.slice/crio-eef4a412512cac98be531503da2caffe3a2d8ea498c18d2993455c5fc172230d WatchSource:0}: Error finding container eef4a412512cac98be531503da2caffe3a2d8ea498c18d2993455c5fc172230d: Status 404 returned error can't find the container with id eef4a412512cac98be531503da2caffe3a2d8ea498c18d2993455c5fc172230d Jul 15 08:09:22.201952 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:22.201927 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:22.203107 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:22.203075 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cdce7e_1bef_4f0f_aa71_3bef566438e3.slice/crio-ab78406c402f7758e93bf08e7e7e41d005515020d0dac238ec867bc4ca2581dc WatchSource:0}: Error finding container ab78406c402f7758e93bf08e7e7e41d005515020d0dac238ec867bc4ca2581dc: Status 404 returned error can't find the container with id ab78406c402f7758e93bf08e7e7e41d005515020d0dac238ec867bc4ca2581dc Jul 15 08:09:22.630445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:22.630393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-xjq7b" event={"ID":"1b388cfc-9218-4bc8-931e-c2653aa9d464","Type":"ContainerStarted","Data":"eef4a412512cac98be531503da2caffe3a2d8ea498c18d2993455c5fc172230d"} Jul 15 08:09:22.631554 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:22.631525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd86fd965-825js" event={"ID":"f1cdce7e-1bef-4f0f-aa71-3bef566438e3","Type":"ContainerStarted","Data":"ab78406c402f7758e93bf08e7e7e41d005515020d0dac238ec867bc4ca2581dc"} Jul 15 08:09:25.093170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.093141 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 08:09:25.650523 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.650480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-xjq7b" event={"ID":"1b388cfc-9218-4bc8-931e-c2653aa9d464","Type":"ContainerStarted","Data":"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19"} Jul 15 08:09:25.650692 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.650556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:25.652035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.652009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd86fd965-825js" event={"ID":"f1cdce7e-1bef-4f0f-aa71-3bef566438e3","Type":"ContainerStarted","Data":"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff"} Jul 15 08:09:25.652168 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.652116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:25.699675 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.699618 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-599b49958f-xjq7b" podStartSLOduration=1.7759856219999999 podStartE2EDuration="4.699603592s" totalImagesPullingTime="2.92361797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:09:25.697240572 +0000 UTC m=+543.116985388" watchObservedRunningTime="2026-07-15 08:09:25.699603592 +0000 UTC m=+543.119348395" Jul 15 08:09:25.737198 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:25.737147 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dd86fd965-825js" podStartSLOduration=1.851311387 podStartE2EDuration="4.737132007s" totalImagesPullingTime="2.88582062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:09:25.736530484 +0000 UTC m=+543.156275288" watchObservedRunningTime="2026-07-15 08:09:25.737132007 +0000 UTC m=+543.156876810" Jul 15 08:09:36.658803 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:36.658770 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:09:36.659364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:36.658968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:36.725251 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:36.725206 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:36.725537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:36.725507 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dd86fd965-825js" podUID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" containerName="manager" containerID="cri-o://e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff" gracePeriod=30 Jul 15 08:09:36.968628 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:36.968605 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:37.110446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.110389 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj\") pod \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " Jul 15 08:09:37.110446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.110456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert\") pod \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\" (UID: \"f1cdce7e-1bef-4f0f-aa71-3bef566438e3\") " Jul 15 08:09:37.112681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.112652 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f1cdce7e-1bef-4f0f-aa71-3bef566438e3" (UID: "f1cdce7e-1bef-4f0f-aa71-3bef566438e3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:09:37.112681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.112669 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj" (OuterVolumeSpecName: "kube-api-access-496wj") pod "f1cdce7e-1bef-4f0f-aa71-3bef566438e3" (UID: "f1cdce7e-1bef-4f0f-aa71-3bef566438e3"). InnerVolumeSpecName "kube-api-access-496wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:09:37.211816 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.211723 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-kube-api-access-496wj\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:09:37.211816 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.211765 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1cdce7e-1bef-4f0f-aa71-3bef566438e3-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:09:37.711725 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.711691 2565 generic.go:356] "Generic (PLEG): container finished" podID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" containerID="e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff" exitCode=0 Jul 15 08:09:37.712170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.711739 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd86fd965-825js" Jul 15 08:09:37.712170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.711778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd86fd965-825js" event={"ID":"f1cdce7e-1bef-4f0f-aa71-3bef566438e3","Type":"ContainerDied","Data":"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff"} Jul 15 08:09:37.712170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.711829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd86fd965-825js" event={"ID":"f1cdce7e-1bef-4f0f-aa71-3bef566438e3","Type":"ContainerDied","Data":"ab78406c402f7758e93bf08e7e7e41d005515020d0dac238ec867bc4ca2581dc"} Jul 15 08:09:37.712170 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.711851 2565 scope.go:117] "RemoveContainer" containerID="e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff" Jul 15 08:09:37.720125 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.720107 2565 scope.go:117] "RemoveContainer" containerID="e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff" Jul 15 08:09:37.720383 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:09:37.720352 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff\": container with ID starting with e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff not found: ID does not exist" containerID="e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff" Jul 15 08:09:37.720489 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.720391 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff"} err="failed to get container status \"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff\": rpc error: code = NotFound desc = could not find container \"e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff\": container with ID starting with e70b80b92a777746c9c3f8807ef6c1bab89885f7777d4667d2486e0d17486cff not found: ID does not exist" Jul 15 08:09:37.730267 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.730243 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:37.734715 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:37.734694 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dd86fd965-825js"] Jul 15 08:09:39.116581 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:39.116550 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" path="/var/lib/kubelet/pods/f1cdce7e-1bef-4f0f-aa71-3bef566438e3/volumes" Jul 15 08:09:57.628043 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.628006 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf"] Jul 15 08:09:57.628634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.628609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" containerName="manager" Jul 15 08:09:57.628634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.628629 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" containerName="manager" Jul 15 08:09:57.628756 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.628712 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1cdce7e-1bef-4f0f-aa71-3bef566438e3" containerName="manager" Jul 15 08:09:57.631885 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.631865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.634222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.634200 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 08:09:57.634325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.634293 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cd6kw\"" Jul 15 08:09:57.641344 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.641310 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf"] Jul 15 08:09:57.703133 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.703104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3deee17-c3de-48f3-9487-097c0fb06aea-openshift-service-ca\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.703288 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.703142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.703288 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.703160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7bln\" (UniqueName: \"kubernetes.io/projected/d3deee17-c3de-48f3-9487-097c0fb06aea-kube-api-access-l7bln\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.804045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.804013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3deee17-c3de-48f3-9487-097c0fb06aea-openshift-service-ca\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.804244 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.804049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.804244 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.804075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bln\" (UniqueName: \"kubernetes.io/projected/d3deee17-c3de-48f3-9487-097c0fb06aea-kube-api-access-l7bln\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.804244 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:09:57.804169 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 15 08:09:57.804244 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:09:57.804220 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls podName:d3deee17-c3de-48f3-9487-097c0fb06aea nodeName:}" failed. No retries permitted until 2026-07-15 08:09:58.304205745 +0000 UTC m=+575.723950527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls") pod "maas-api-b5b9647fd-ncrnf" (UID: "d3deee17-c3de-48f3-9487-097c0fb06aea") : secret "maas-api-serving-cert" not found Jul 15 08:09:57.804803 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.804779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3deee17-c3de-48f3-9487-097c0fb06aea-openshift-service-ca\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:57.812987 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:57.812962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bln\" (UniqueName: \"kubernetes.io/projected/d3deee17-c3de-48f3-9487-097c0fb06aea-kube-api-access-l7bln\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:58.180358 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.180324 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-87ng7"] Jul 15 08:09:58.187832 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.187811 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.190353 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.190324 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2dbbh\"" Jul 15 08:09:58.190647 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.190620 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 08:09:58.192182 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.192159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-87ng7"] Jul 15 08:09:58.209713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.209687 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-blmdw"] Jul 15 08:09:58.214534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.214517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.224430 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.224394 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-blmdw"] Jul 15 08:09:58.310031 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.309999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2059bfc-ed4a-4261-bad1-36996b060389-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.310192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.310102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6j27\" (UniqueName: \"kubernetes.io/projected/a2059bfc-ed4a-4261-bad1-36996b060389-kube-api-access-d6j27\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.310192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.310122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e58d70eb-1088-4c87-91d1-bd5a791b0aba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.310192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.310169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:58.310323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.310203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx955\" (UniqueName: \"kubernetes.io/projected/e58d70eb-1088-4c87-91d1-bd5a791b0aba-kube-api-access-rx955\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.312471 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.312453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3deee17-c3de-48f3-9487-097c0fb06aea-maas-api-tls\") pod \"maas-api-b5b9647fd-ncrnf\" (UID: \"d3deee17-c3de-48f3-9487-097c0fb06aea\") " pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:58.411456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.411394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6j27\" (UniqueName: \"kubernetes.io/projected/a2059bfc-ed4a-4261-bad1-36996b060389-kube-api-access-d6j27\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.411619 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.411472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e58d70eb-1088-4c87-91d1-bd5a791b0aba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.411619 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.411537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx955\" (UniqueName: \"kubernetes.io/projected/e58d70eb-1088-4c87-91d1-bd5a791b0aba-kube-api-access-rx955\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.411619 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.411607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2059bfc-ed4a-4261-bad1-36996b060389-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.412364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.412342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2059bfc-ed4a-4261-bad1-36996b060389-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.412364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.412355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e58d70eb-1088-4c87-91d1-bd5a791b0aba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.420299 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.420278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx955\" (UniqueName: \"kubernetes.io/projected/e58d70eb-1088-4c87-91d1-bd5a791b0aba-kube-api-access-rx955\") pod \"payload-processing-f9b8d9b9b-blmdw\" (UID: \"e58d70eb-1088-4c87-91d1-bd5a791b0aba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.420459 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.420438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6j27\" (UniqueName: \"kubernetes.io/projected/a2059bfc-ed4a-4261-bad1-36996b060389-kube-api-access-d6j27\") pod \"payload-pre-processing-74584bb9f8-87ng7\" (UID: \"a2059bfc-ed4a-4261-bad1-36996b060389\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.499731 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.499702 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:09:58.525019 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.524991 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:09:58.542254 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.542222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:09:58.657556 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.657336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-87ng7"] Jul 15 08:09:58.660628 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:58.660591 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2059bfc_ed4a_4261_bad1_36996b060389.slice/crio-aad358b47e7d07700e7edf3d35c18108067670ac0363f33984adc7baea8ac10a WatchSource:0}: Error finding container aad358b47e7d07700e7edf3d35c18108067670ac0363f33984adc7baea8ac10a: Status 404 returned error can't find the container with id aad358b47e7d07700e7edf3d35c18108067670ac0363f33984adc7baea8ac10a Jul 15 08:09:58.687470 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.687432 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-blmdw"] Jul 15 08:09:58.691470 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:58.691441 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58d70eb_1088_4c87_91d1_bd5a791b0aba.slice/crio-c9fdb2eed4eef7db81a6b6f0bf948479bfaa7966c38c39dd45d8b5084f7fd9f0 WatchSource:0}: Error finding container c9fdb2eed4eef7db81a6b6f0bf948479bfaa7966c38c39dd45d8b5084f7fd9f0: Status 404 returned error can't find the container with id c9fdb2eed4eef7db81a6b6f0bf948479bfaa7966c38c39dd45d8b5084f7fd9f0 Jul 15 08:09:58.710962 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.710938 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf"] Jul 15 08:09:58.712588 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:09:58.712563 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3deee17_c3de_48f3_9487_097c0fb06aea.slice/crio-f3c3fe7cb24ab553693292cf19ab46e6b609a9f83f5ac9495b054caf60217421 WatchSource:0}: Error finding container f3c3fe7cb24ab553693292cf19ab46e6b609a9f83f5ac9495b054caf60217421: Status 404 returned error can't find the container with id f3c3fe7cb24ab553693292cf19ab46e6b609a9f83f5ac9495b054caf60217421 Jul 15 08:09:58.819535 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.819433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" event={"ID":"e58d70eb-1088-4c87-91d1-bd5a791b0aba","Type":"ContainerStarted","Data":"c9fdb2eed4eef7db81a6b6f0bf948479bfaa7966c38c39dd45d8b5084f7fd9f0"} Jul 15 08:09:58.820827 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.820803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" event={"ID":"d3deee17-c3de-48f3-9487-097c0fb06aea","Type":"ContainerStarted","Data":"f3c3fe7cb24ab553693292cf19ab46e6b609a9f83f5ac9495b054caf60217421"} Jul 15 08:09:58.822002 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:09:58.821975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" event={"ID":"a2059bfc-ed4a-4261-bad1-36996b060389","Type":"ContainerStarted","Data":"aad358b47e7d07700e7edf3d35c18108067670ac0363f33984adc7baea8ac10a"} Jul 15 08:10:00.835586 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.835552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" event={"ID":"e58d70eb-1088-4c87-91d1-bd5a791b0aba","Type":"ContainerStarted","Data":"fa7dde1860e4101d4e4c66e44a0ff29f5e487be263b3c961b031e2d72f74f026"} Jul 15 08:10:00.836000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.835644 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:10:00.837188 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.837167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" event={"ID":"d3deee17-c3de-48f3-9487-097c0fb06aea","Type":"ContainerStarted","Data":"3fcfa9a8ee7123fbacfb81c51034268f9c2c3ad264a2be1db9297ca87d082d38"} Jul 15 08:10:00.837287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.837217 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:10:00.838527 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.838505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" event={"ID":"a2059bfc-ed4a-4261-bad1-36996b060389","Type":"ContainerStarted","Data":"8960806b16e51766fe2aaff1e04587c0ecad737bea428854a56974c1070b1687"} Jul 15 08:10:00.838637 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.838623 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:10:00.855730 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.855684 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" podStartSLOduration=0.848896918 podStartE2EDuration="2.855664206s" totalImagesPullingTime="2.006767288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:10:00.852971955 +0000 UTC m=+578.272716792" watchObservedRunningTime="2026-07-15 08:10:00.855664206 +0000 UTC m=+578.275409008" Jul 15 08:10:00.869480 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.869441 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" podStartSLOduration=0.829746298 podStartE2EDuration="2.869425717s" totalImagesPullingTime="2.039679419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:10:00.868103343 +0000 UTC m=+578.287848157" watchObservedRunningTime="2026-07-15 08:10:00.869425717 +0000 UTC m=+578.289170516" Jul 15 08:10:00.885885 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:00.885833 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" podStartSLOduration=1.8997455570000001 podStartE2EDuration="3.885815894s" totalImagesPullingTime="1.986070337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:10:00.884701225 +0000 UTC m=+578.304446030" watchObservedRunningTime="2026-07-15 08:10:00.885815894 +0000 UTC m=+578.305560702" Jul 15 08:10:06.848993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:06.848959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b5b9647fd-ncrnf" Jul 15 08:10:11.844813 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:11.844769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-blmdw" Jul 15 08:10:11.845275 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:11.844877 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-87ng7" Jul 15 08:10:23.072255 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:23.072228 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:10:23.074306 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:23.074287 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:10:23.079533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:23.079516 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:10:23.081688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:10:23.081673 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:11:04.412875 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.412836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5"] Jul 15 08:11:04.417245 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.417228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.419627 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.419607 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 08:11:04.420336 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.420317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 15 08:11:04.420446 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.420372 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 08:11:04.420675 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.420656 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-schk4\"" Jul 15 08:11:04.425794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.425773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5"] Jul 15 08:11:04.512885 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.512853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff1151-115d-4607-bf0d-8fd1747b5126-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.513043 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.512914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/f7ff1151-115d-4607-bf0d-8fd1747b5126-kube-api-access-cvqlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.513043 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.512987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.513121 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.513061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.513121 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.513095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.513121 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.513110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614252 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614442 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614442 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614442 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff1151-115d-4607-bf0d-8fd1747b5126-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614442 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/f7ff1151-115d-4607-bf0d-8fd1747b5126-kube-api-access-cvqlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614597 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614729 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614768 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.614819 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.614805 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.616655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.616636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff1151-115d-4607-bf0d-8fd1747b5126-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.616748 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.616727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff1151-115d-4607-bf0d-8fd1747b5126-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.623499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.623476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/f7ff1151-115d-4607-bf0d-8fd1747b5126-kube-api-access-cvqlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vz2k5\" (UID: \"f7ff1151-115d-4607-bf0d-8fd1747b5126\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.730148 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.730120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:04.862905 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.862870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5"] Jul 15 08:11:04.864598 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:11:04.864563 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff1151_115d_4607_bf0d_8fd1747b5126.slice/crio-71727759eea9ac13636e495214adb06ee76c567a9800209245e6094e0493a815 WatchSource:0}: Error finding container 71727759eea9ac13636e495214adb06ee76c567a9800209245e6094e0493a815: Status 404 returned error can't find the container with id 71727759eea9ac13636e495214adb06ee76c567a9800209245e6094e0493a815 Jul 15 08:11:04.866276 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:04.866259 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:11:05.147877 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:05.147786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" event={"ID":"f7ff1151-115d-4607-bf0d-8fd1747b5126","Type":"ContainerStarted","Data":"71727759eea9ac13636e495214adb06ee76c567a9800209245e6094e0493a815"} Jul 15 08:11:06.154610 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:06.154575 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" event={"ID":"f7ff1151-115d-4607-bf0d-8fd1747b5126","Type":"ContainerStarted","Data":"f9a1912a2f923e8a95d0315d0b0b821edb284698f6850190494ada60b0d8ebbe"} Jul 15 08:11:06.173162 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:06.173074 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" podStartSLOduration=1.120969546 podStartE2EDuration="2.173060429s" totalImagesPullingTime="1.052090883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:11:06.171859012 +0000 UTC m=+643.591603815" watchObservedRunningTime="2026-07-15 08:11:06.173060429 +0000 UTC m=+643.592805236" Jul 15 08:11:08.337456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.337422 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx"] Jul 15 08:11:08.341064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.341043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.343644 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.343626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 08:11:08.351285 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.351264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx"] Jul 15 08:11:08.353064 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.353172 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.353172 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.353265 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.353303 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.353345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.353329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8782n\" (UniqueName: \"kubernetes.io/projected/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-kube-api-access-8782n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.453880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.453849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.453888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.453949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.453988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8782n\" (UniqueName: \"kubernetes.io/projected/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-kube-api-access-8782n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.454017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454249 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.454048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454360 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.454339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.454421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.454456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.454442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.455968 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.455950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.456290 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.456272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.462395 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.462373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8782n\" (UniqueName: \"kubernetes.io/projected/651da742-7e85-4eb1-8b44-b3fa5b4cdc86-kube-api-access-8782n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggqqx\" (UID: \"651da742-7e85-4eb1-8b44-b3fa5b4cdc86\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.651957 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.651877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:08.783203 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:08.783175 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx"] Jul 15 08:11:08.785731 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:11:08.785702 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651da742_7e85_4eb1_8b44_b3fa5b4cdc86.slice/crio-ea90782b883a7dfdf2f9d37a846eaa2ae26f5ded1c229331a035a8bf6738a218 WatchSource:0}: Error finding container ea90782b883a7dfdf2f9d37a846eaa2ae26f5ded1c229331a035a8bf6738a218: Status 404 returned error can't find the container with id ea90782b883a7dfdf2f9d37a846eaa2ae26f5ded1c229331a035a8bf6738a218 Jul 15 08:11:09.170856 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:09.170820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" event={"ID":"651da742-7e85-4eb1-8b44-b3fa5b4cdc86","Type":"ContainerStarted","Data":"e972dcbc60873e97d23cab4fce930e73570751e3804f3aba416e9163d6d60b95"} Jul 15 08:11:09.171034 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:09.170876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" event={"ID":"651da742-7e85-4eb1-8b44-b3fa5b4cdc86","Type":"ContainerStarted","Data":"ea90782b883a7dfdf2f9d37a846eaa2ae26f5ded1c229331a035a8bf6738a218"} Jul 15 08:11:09.188019 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:09.187971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" podStartSLOduration=0.955883193 podStartE2EDuration="1.187956171s" totalImagesPullingTime="232.072978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:11:09.186205315 +0000 UTC m=+646.605950119" watchObservedRunningTime="2026-07-15 08:11:09.187956171 +0000 UTC m=+646.607700975" Jul 15 08:11:14.731016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:14.730983 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:14.731373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:14.731045 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:14.743503 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:14.743483 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:15.209194 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:15.209116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vz2k5" Jul 15 08:11:18.652031 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:18.651995 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:18.652469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:18.652047 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:18.664390 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:18.664366 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:11:19.227583 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:11:19.227558 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggqqx" Jul 15 08:13:55.284325 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.284245 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:13:55.284898 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.284451 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-599b49958f-xjq7b" podUID="1b388cfc-9218-4bc8-931e-c2653aa9d464" containerName="manager" containerID="cri-o://601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19" gracePeriod=30 Jul 15 08:13:55.538460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.538365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:13:55.668924 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.668887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert\") pod \"1b388cfc-9218-4bc8-931e-c2653aa9d464\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " Jul 15 08:13:55.669097 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.668965 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt7p5\" (UniqueName: \"kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5\") pod \"1b388cfc-9218-4bc8-931e-c2653aa9d464\" (UID: \"1b388cfc-9218-4bc8-931e-c2653aa9d464\") " Jul 15 08:13:55.670963 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.670936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5" (OuterVolumeSpecName: "kube-api-access-bt7p5") pod "1b388cfc-9218-4bc8-931e-c2653aa9d464" (UID: "1b388cfc-9218-4bc8-931e-c2653aa9d464"). InnerVolumeSpecName "kube-api-access-bt7p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:13:55.670963 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.670936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1b388cfc-9218-4bc8-931e-c2653aa9d464" (UID: "1b388cfc-9218-4bc8-931e-c2653aa9d464"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:13:55.770724 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.770691 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b388cfc-9218-4bc8-931e-c2653aa9d464-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:13:55.770892 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.770742 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt7p5\" (UniqueName: \"kubernetes.io/projected/1b388cfc-9218-4bc8-931e-c2653aa9d464-kube-api-access-bt7p5\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:13:55.968773 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.968678 2565 generic.go:356] "Generic (PLEG): container finished" podID="1b388cfc-9218-4bc8-931e-c2653aa9d464" containerID="601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19" exitCode=0 Jul 15 08:13:55.968773 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.968727 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-xjq7b" Jul 15 08:13:55.968773 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.968758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-xjq7b" event={"ID":"1b388cfc-9218-4bc8-931e-c2653aa9d464","Type":"ContainerDied","Data":"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19"} Jul 15 08:13:55.969001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.968841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-xjq7b" event={"ID":"1b388cfc-9218-4bc8-931e-c2653aa9d464","Type":"ContainerDied","Data":"eef4a412512cac98be531503da2caffe3a2d8ea498c18d2993455c5fc172230d"} Jul 15 08:13:55.969001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.968863 2565 scope.go:117] "RemoveContainer" containerID="601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19" Jul 15 08:13:55.978074 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.978057 2565 scope.go:117] "RemoveContainer" containerID="601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19" Jul 15 08:13:55.978317 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:13:55.978301 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19\": container with ID starting with 601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19 not found: ID does not exist" containerID="601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19" Jul 15 08:13:55.978363 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.978324 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19"} err="failed to get container status \"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19\": rpc error: code = NotFound desc = could not find container \"601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19\": container with ID starting with 601c818e9cb65af6f1687bb0f640bb6896333a348fba5f676293a38c17f8bf19 not found: ID does not exist" Jul 15 08:13:55.990904 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.990876 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:13:55.996813 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:55.996793 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-599b49958f-xjq7b"] Jul 15 08:13:56.825079 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.825044 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-599b49958f-6q29h"] Jul 15 08:13:56.825482 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.825459 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b388cfc-9218-4bc8-931e-c2653aa9d464" containerName="manager" Jul 15 08:13:56.825482 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.825481 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b388cfc-9218-4bc8-931e-c2653aa9d464" containerName="manager" Jul 15 08:13:56.825575 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.825560 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b388cfc-9218-4bc8-931e-c2653aa9d464" containerName="manager" Jul 15 08:13:56.830033 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.830012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:56.833244 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.833225 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 08:13:56.833361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.833282 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 08:13:56.833361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.833282 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6wnqs\"" Jul 15 08:13:56.838755 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.838737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-599b49958f-6q29h"] Jul 15 08:13:56.982898 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.982862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhcd\" (UniqueName: \"kubernetes.io/projected/43489291-da25-4a70-9791-abf63f8c7c0a-kube-api-access-lnhcd\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:56.983077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:56.982918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43489291-da25-4a70-9791-abf63f8c7c0a-webhook-cert\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.083666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.083582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43489291-da25-4a70-9791-abf63f8c7c0a-webhook-cert\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.083830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.083685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhcd\" (UniqueName: \"kubernetes.io/projected/43489291-da25-4a70-9791-abf63f8c7c0a-kube-api-access-lnhcd\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.086068 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.086043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43489291-da25-4a70-9791-abf63f8c7c0a-webhook-cert\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.092869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.092842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhcd\" (UniqueName: \"kubernetes.io/projected/43489291-da25-4a70-9791-abf63f8c7c0a-kube-api-access-lnhcd\") pod \"maas-controller-599b49958f-6q29h\" (UID: \"43489291-da25-4a70-9791-abf63f8c7c0a\") " pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.116460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.116429 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b388cfc-9218-4bc8-931e-c2653aa9d464" path="/var/lib/kubelet/pods/1b388cfc-9218-4bc8-931e-c2653aa9d464/volumes" Jul 15 08:13:57.140561 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.140528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:57.478383 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.478357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-599b49958f-6q29h"] Jul 15 08:13:57.480179 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:13:57.480138 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43489291_da25_4a70_9791_abf63f8c7c0a.slice/crio-28f36b11f4b27d0af4836b84ae708068da432a5ce23d09490e2e81184299e16b WatchSource:0}: Error finding container 28f36b11f4b27d0af4836b84ae708068da432a5ce23d09490e2e81184299e16b: Status 404 returned error can't find the container with id 28f36b11f4b27d0af4836b84ae708068da432a5ce23d09490e2e81184299e16b Jul 15 08:13:57.980972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:57.980949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-6q29h" event={"ID":"43489291-da25-4a70-9791-abf63f8c7c0a","Type":"ContainerStarted","Data":"28f36b11f4b27d0af4836b84ae708068da432a5ce23d09490e2e81184299e16b"} Jul 15 08:13:58.987078 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:58.987048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-599b49958f-6q29h" event={"ID":"43489291-da25-4a70-9791-abf63f8c7c0a","Type":"ContainerStarted","Data":"65fb5af3b971ef99e866291f9d4ce918968a7ba7b512f7e13f30753a1fcb618f"} Jul 15 08:13:58.987518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:58.987119 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:13:59.005601 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:13:59.005555 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-599b49958f-6q29h" podStartSLOduration=2.5494413 podStartE2EDuration="3.005540883s" totalImagesPullingTime="456.099583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:13:59.003737014 +0000 UTC m=+816.423481818" watchObservedRunningTime="2026-07-15 08:13:59.005540883 +0000 UTC m=+816.425285687" Jul 15 08:14:09.994823 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:14:09.994794 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-599b49958f-6q29h" Jul 15 08:15:00.172203 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.172164 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm"] Jul 15 08:15:00.175488 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.175469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" Jul 15 08:15:00.180077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.180058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:15:00.184869 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.184852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" Jul 15 08:15:00.195512 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.195473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm"] Jul 15 08:15:00.329959 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:00.329937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm"] Jul 15 08:15:00.332650 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:15:00.332619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5357d86a_d793_492b_b10e_7559572cc2cb.slice/crio-3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758 WatchSource:0}: Error finding container 3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758: Status 404 returned error can't find the container with id 3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758 Jul 15 08:15:01.288900 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:01.288863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" event={"ID":"5357d86a-d793-492b-b10e-7559572cc2cb","Type":"ContainerStarted","Data":"3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758"} Jul 15 08:15:03.300281 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:03.300251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" event={"ID":"5357d86a-d793-492b-b10e-7559572cc2cb","Type":"ContainerStarted","Data":"3198045d309fceb10c0c1899e8bdbb8fb3793a00d16c205c8b146ab2e843d3ab"} Jul 15 08:15:03.319911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:03.319860 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" podStartSLOduration=1.106127286 podStartE2EDuration="3.319842041s" totalImagesPullingTime="2.213714755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:15:03.318613809 +0000 UTC m=+880.738358624" watchObservedRunningTime="2026-07-15 08:15:03.319842041 +0000 UTC m=+880.739586844" Jul 15 08:15:04.306648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:04.306614 2565 generic.go:356] "Generic (PLEG): container finished" podID="5357d86a-d793-492b-b10e-7559572cc2cb" containerID="3198045d309fceb10c0c1899e8bdbb8fb3793a00d16c205c8b146ab2e843d3ab" exitCode=0 Jul 15 08:15:04.307077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:04.306696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" event={"ID":"5357d86a-d793-492b-b10e-7559572cc2cb","Type":"ContainerDied","Data":"3198045d309fceb10c0c1899e8bdbb8fb3793a00d16c205c8b146ab2e843d3ab"} Jul 15 08:15:05.450840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:05.450819 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" Jul 15 08:15:06.317533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:06.317491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" event={"ID":"5357d86a-d793-492b-b10e-7559572cc2cb","Type":"ContainerDied","Data":"3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758"} Jul 15 08:15:06.317533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:06.317512 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735055-n8rcm" Jul 15 08:15:06.317899 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:06.317543 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eef2889e2824f22d068cd2ac6c9aa4d8c535e8d2b4308392d015738a2ec7758" Jul 15 08:15:23.110688 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:23.110658 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:15:23.111200 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:23.110764 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:15:23.118721 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:23.118699 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:15:23.118859 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:15:23.118829 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:20:23.142175 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:20:23.142141 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:20:23.144307 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:20:23.144285 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:20:23.149192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:20:23.149172 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:20:23.151295 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:20:23.151275 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:25:23.174268 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:23.174241 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:25:23.178038 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:23.178018 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:25:23.181594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:23.181579 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:25:23.184793 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:23.184778 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:25:50.975564 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.975531 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:25:50.975951 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.975938 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5357d86a-d793-492b-b10e-7559572cc2cb" containerName="cleanup" Jul 15 08:25:50.975993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.975952 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5357d86a-d793-492b-b10e-7559572cc2cb" containerName="cleanup" Jul 15 08:25:50.976027 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.976016 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5357d86a-d793-492b-b10e-7559572cc2cb" containerName="cleanup" Jul 15 08:25:50.979279 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.979262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:50.983664 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:50.983645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f0ce8ae8-serving-cert\"" Jul 15 08:25:51.004285 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.004257 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:25:51.079297 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.079265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.079297 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.079302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9m2\" (UniqueName: \"kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.079533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.079435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.180239 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.180206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.180432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.180372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.180432 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.180391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9m2\" (UniqueName: \"kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.181017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.180998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.182746 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.182719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.191287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.191262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9m2\" (UniqueName: \"kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2\") pod \"maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.289159 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.289127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:51.440802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.440777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:25:51.443265 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:25:51.443237 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54609c7_a905_4caa_9fe7_cbd91452c2fd.slice/crio-1915826e6bac7a07fd0b6516ca344cd1066c611dc0d470cc24c4bb172852b49b WatchSource:0}: Error finding container 1915826e6bac7a07fd0b6516ca344cd1066c611dc0d470cc24c4bb172852b49b: Status 404 returned error can't find the container with id 1915826e6bac7a07fd0b6516ca344cd1066c611dc0d470cc24c4bb172852b49b Jul 15 08:25:51.444855 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.444835 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:25:51.526721 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:51.526683 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" event={"ID":"a54609c7-a905-4caa-9fe7-cbd91452c2fd","Type":"ContainerStarted","Data":"1915826e6bac7a07fd0b6516ca344cd1066c611dc0d470cc24c4bb172852b49b"} Jul 15 08:25:52.533514 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:52.533480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" event={"ID":"a54609c7-a905-4caa-9fe7-cbd91452c2fd","Type":"ContainerStarted","Data":"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956"} Jul 15 08:25:52.533897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:52.533587 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:52.555644 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:52.555595 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" podStartSLOduration=2.066423141 podStartE2EDuration="2.555579194s" totalImagesPullingTime="489.156053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:25:52.553703068 +0000 UTC m=+1529.973447869" watchObservedRunningTime="2026-07-15 08:25:52.555579194 +0000 UTC m=+1529.975323996" Jul 15 08:25:55.868557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:55.868516 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:25:55.872704 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:55.872685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:25:55.875164 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:55.875145 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:25:55.878766 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:55.878741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:25:55.927300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:55.927262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj\" (UID: \"9a11ebb6-9df2-4b71-8868-cb3245f37ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:25:56.028238 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:56.028200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj\" (UID: \"9a11ebb6-9df2-4b71-8868-cb3245f37ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:25:56.028822 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:56.028803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj\" (UID: \"9a11ebb6-9df2-4b71-8868-cb3245f37ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:25:56.185045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:56.184958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:25:56.320757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:56.320728 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:25:56.323612 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:25:56.323582 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a11ebb6_9df2_4b71_8868_cb3245f37ac5.slice/crio-532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff WatchSource:0}: Error finding container 532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff: Status 404 returned error can't find the container with id 532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff Jul 15 08:25:56.555021 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:56.554987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerStarted","Data":"532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff"} Jul 15 08:25:57.561646 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:57.561608 2565 generic.go:356] "Generic (PLEG): container finished" podID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerID="b71b5d3c5cd97a3ecbbc8f81ebbf0573830e52c9de3b76e1c70850dad25751d9" exitCode=7 Jul 15 08:25:57.562091 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:57.561699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerDied","Data":"b71b5d3c5cd97a3ecbbc8f81ebbf0573830e52c9de3b76e1c70850dad25751d9"} Jul 15 08:25:57.562091 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:57.561850 2565 scope.go:117] "RemoveContainer" containerID="b71b5d3c5cd97a3ecbbc8f81ebbf0573830e52c9de3b76e1c70850dad25751d9" Jul 15 08:25:58.544991 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:58.544959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:25:58.569570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:58.569532 2565 generic.go:356] "Generic (PLEG): container finished" podID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerID="2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104" exitCode=7 Jul 15 08:25:58.570017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:58.569616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerDied","Data":"2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104"} Jul 15 08:25:58.570017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:58.569696 2565 scope.go:117] "RemoveContainer" containerID="b71b5d3c5cd97a3ecbbc8f81ebbf0573830e52c9de3b76e1c70850dad25751d9" Jul 15 08:25:58.570017 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:58.569881 2565 scope.go:117] "RemoveContainer" containerID="2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104" Jul 15 08:25:58.570210 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:25:58.570184 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj_odh-ai-gateway-infra(9a11ebb6-9df2-4b71-8868-cb3245f37ac5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" Jul 15 08:25:59.577572 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:25:59.577545 2565 scope.go:117] "RemoveContainer" containerID="2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104" Jul 15 08:25:59.577961 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:25:59.577769 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj_odh-ai-gateway-infra(9a11ebb6-9df2-4b71-8868-cb3245f37ac5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" Jul 15 08:26:14.111124 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:14.111090 2565 scope.go:117] "RemoveContainer" containerID="2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104" Jul 15 08:26:14.659132 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:14.659097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerStarted","Data":"34a32f260b89930a0114115bdda73155da9e05a7fe29b69eb7f9436f8bf26786"} Jul 15 08:26:14.679447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:14.679381 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" podStartSLOduration=19.679365552 podStartE2EDuration="19.679365552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:26:14.677307099 +0000 UTC m=+1552.097051913" watchObservedRunningTime="2026-07-15 08:26:14.679365552 +0000 UTC m=+1552.099110353" Jul 15 08:26:15.666229 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:15.666196 2565 generic.go:356] "Generic (PLEG): container finished" podID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerID="34a32f260b89930a0114115bdda73155da9e05a7fe29b69eb7f9436f8bf26786" exitCode=0 Jul 15 08:26:15.666683 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:15.666275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerDied","Data":"34a32f260b89930a0114115bdda73155da9e05a7fe29b69eb7f9436f8bf26786"} Jul 15 08:26:15.666683 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:15.666342 2565 scope.go:117] "RemoveContainer" containerID="2cfa9b61bdba9ed9262092ad65d72f1abb3cba4292e1e1e163de9c4ae755a104" Jul 15 08:26:15.689180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:15.689147 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:26:16.813686 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:16.813658 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:26:16.836094 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:16.836057 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca\") pod \"9a11ebb6-9df2-4b71-8868-cb3245f37ac5\" (UID: \"9a11ebb6-9df2-4b71-8868-cb3245f37ac5\") " Jul 15 08:26:16.836374 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:16.836355 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a11ebb6-9df2-4b71-8868-cb3245f37ac5" (UID: "9a11ebb6-9df2-4b71-8868-cb3245f37ac5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:26:16.836471 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:16.836453 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a11ebb6-9df2-4b71-8868-cb3245f37ac5-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:26:17.678902 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:17.678874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" Jul 15 08:26:17.678902 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:17.678884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj" event={"ID":"9a11ebb6-9df2-4b71-8868-cb3245f37ac5","Type":"ContainerDied","Data":"532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff"} Jul 15 08:26:17.679195 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:17.678929 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532395e5ccc0025d5e69c76463eb73509880ffdfc37846996346645271236aff" Jul 15 08:26:17.707631 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:17.707600 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:26:17.713895 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:17.713869 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-km5dj"] Jul 15 08:26:19.116891 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:19.116857 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" path="/var/lib/kubelet/pods/9a11ebb6-9df2-4b71-8868-cb3245f37ac5/volumes" Jul 15 08:26:55.915182 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc"] Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915571 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915582 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915596 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915601 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915677 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.915802 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.915688 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:26:55.918764 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.918744 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:55.921336 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.921316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:26:55.927354 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:55.927329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc"] Jul 15 08:26:56.015971 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.015931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc\" (UID: \"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:56.116564 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.116525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc\" (UID: \"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:56.117148 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.117130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc\" (UID: \"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:56.230741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.230715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:56.376895 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.369023 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc"] Jul 15 08:26:56.888809 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.888709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" event={"ID":"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d","Type":"ContainerStarted","Data":"1f71116e7d917814eb9f5df909f02abdbaf3fd63f33fc40d293f03f879627ef4"} Jul 15 08:26:56.888809 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.888777 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" event={"ID":"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d","Type":"ContainerStarted","Data":"d4a66a5b456fbc19b06d442cfc9014e4fe0034adfbfb5f5bce8d26a3232910c8"} Jul 15 08:26:56.907831 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:56.907776 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" podStartSLOduration=1.907762698 podStartE2EDuration="1.907762698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:26:56.90601616 +0000 UTC m=+1594.325760960" watchObservedRunningTime="2026-07-15 08:26:56.907762698 +0000 UTC m=+1594.327507501" Jul 15 08:26:57.894922 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:57.894890 2565 generic.go:356] "Generic (PLEG): container finished" podID="d03d3ba6-cf73-4887-aad7-ea267b6b2c6d" containerID="1f71116e7d917814eb9f5df909f02abdbaf3fd63f33fc40d293f03f879627ef4" exitCode=0 Jul 15 08:26:57.895292 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:57.894957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" event={"ID":"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d","Type":"ContainerDied","Data":"1f71116e7d917814eb9f5df909f02abdbaf3fd63f33fc40d293f03f879627ef4"} Jul 15 08:26:59.035371 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.035347 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:59.145750 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.145717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca\") pod \"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d\" (UID: \"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d\") " Jul 15 08:26:59.146081 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.146059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d03d3ba6-cf73-4887-aad7-ea267b6b2c6d" (UID: "d03d3ba6-cf73-4887-aad7-ea267b6b2c6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:26:59.247467 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.247434 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d3ba6-cf73-4887-aad7-ea267b6b2c6d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:26:59.906102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.906025 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" Jul 15 08:26:59.906242 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.906032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc" event={"ID":"d03d3ba6-cf73-4887-aad7-ea267b6b2c6d","Type":"ContainerDied","Data":"d4a66a5b456fbc19b06d442cfc9014e4fe0034adfbfb5f5bce8d26a3232910c8"} Jul 15 08:26:59.906242 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:26:59.906155 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a66a5b456fbc19b06d442cfc9014e4fe0034adfbfb5f5bce8d26a3232910c8" Jul 15 08:27:06.350810 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.350774 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:27:06.351270 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.350949 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" podUID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" containerName="maas-api" containerID="cri-o://1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956" gracePeriod=30 Jul 15 08:27:06.609051 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.608983 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:27:06.714174 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.714137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd9m2\" (UniqueName: \"kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2\") pod \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " Jul 15 08:27:06.714348 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.714193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca\") pod \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " Jul 15 08:27:06.714348 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.714312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls\") pod \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\" (UID: \"a54609c7-a905-4caa-9fe7-cbd91452c2fd\") " Jul 15 08:27:06.714571 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.714551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a54609c7-a905-4caa-9fe7-cbd91452c2fd" (UID: "a54609c7-a905-4caa-9fe7-cbd91452c2fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:27:06.714703 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.714691 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54609c7-a905-4caa-9fe7-cbd91452c2fd-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:27:06.716460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.716440 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a54609c7-a905-4caa-9fe7-cbd91452c2fd" (UID: "a54609c7-a905-4caa-9fe7-cbd91452c2fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:27:06.716662 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.716639 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2" (OuterVolumeSpecName: "kube-api-access-nd9m2") pod "a54609c7-a905-4caa-9fe7-cbd91452c2fd" (UID: "a54609c7-a905-4caa-9fe7-cbd91452c2fd"). InnerVolumeSpecName "kube-api-access-nd9m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:27:06.815350 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.815313 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9m2\" (UniqueName: \"kubernetes.io/projected/a54609c7-a905-4caa-9fe7-cbd91452c2fd-kube-api-access-nd9m2\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:27:06.815350 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.815347 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54609c7-a905-4caa-9fe7-cbd91452c2fd-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:27:06.949710 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.949618 2565 generic.go:356] "Generic (PLEG): container finished" podID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" containerID="1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956" exitCode=0 Jul 15 08:27:06.949710 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.949658 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" Jul 15 08:27:06.949897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.949704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" event={"ID":"a54609c7-a905-4caa-9fe7-cbd91452c2fd","Type":"ContainerDied","Data":"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956"} Jul 15 08:27:06.949897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.949755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m" event={"ID":"a54609c7-a905-4caa-9fe7-cbd91452c2fd","Type":"ContainerDied","Data":"1915826e6bac7a07fd0b6516ca344cd1066c611dc0d470cc24c4bb172852b49b"} Jul 15 08:27:06.949897 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.949776 2565 scope.go:117] "RemoveContainer" containerID="1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956" Jul 15 08:27:06.959058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.959035 2565 scope.go:117] "RemoveContainer" containerID="1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956" Jul 15 08:27:06.959327 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:27:06.959307 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956\": container with ID starting with 1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956 not found: ID does not exist" containerID="1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956" Jul 15 08:27:06.959379 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.959336 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956"} err="failed to get container status \"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956\": rpc error: code = NotFound desc = could not find container \"1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956\": container with ID starting with 1cf012e836eee1dec39489227fce8e419244cec370c90959a334504ffab42956 not found: ID does not exist" Jul 15 08:27:06.977139 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.977099 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:27:06.981356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:06.981328 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0ce8ae8-8bc8dcffc-djp2m"] Jul 15 08:27:07.116191 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:07.116158 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" path="/var/lib/kubelet/pods/a54609c7-a905-4caa-9fe7-cbd91452c2fd/volumes" Jul 15 08:27:23.151747 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.151709 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:27:23.152360 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152339 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:27:23.152454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152363 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:27:23.152454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152379 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03d3ba6-cf73-4887-aad7-ea267b6b2c6d" containerName="revoke-keys" Jul 15 08:27:23.152454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152388 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03d3ba6-cf73-4887-aad7-ea267b6b2c6d" containerName="revoke-keys" Jul 15 08:27:23.152454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152437 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" containerName="maas-api" Jul 15 08:27:23.152454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152447 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" containerName="maas-api" Jul 15 08:27:23.152728 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152549 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54609c7-a905-4caa-9fe7-cbd91452c2fd" containerName="maas-api" Jul 15 08:27:23.152728 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152563 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a11ebb6-9df2-4b71-8868-cb3245f37ac5" containerName="revoke-keys" Jul 15 08:27:23.152728 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.152576 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03d3ba6-cf73-4887-aad7-ea267b6b2c6d" containerName="revoke-keys" Jul 15 08:27:23.157129 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.157107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.159435 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.159397 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cdca1605-serving-cert\"" Jul 15 08:27:23.164342 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.164321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:27:23.269656 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.269633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.269785 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.269765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7mt\" (UniqueName: \"kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.269830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.269797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.370757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.370730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7mt\" (UniqueName: \"kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.370866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.370762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.370866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.370791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.370978 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:27:23.370960 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-serving-cert: secret "maas-api-e2e-migrate-cdca1605-serving-cert" not found Jul 15 08:27:23.371043 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:27:23.371034 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls podName:c054f45f-4bd4-4f44-a425-766926463348 nodeName:}" failed. No retries permitted until 2026-07-15 08:27:23.871019433 +0000 UTC m=+1621.290764214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls") pod "maas-api-e2e-migrate-cdca1605-976bcb87-7st97" (UID: "c054f45f-4bd4-4f44-a425-766926463348") : secret "maas-api-e2e-migrate-cdca1605-serving-cert" not found Jul 15 08:27:23.371377 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.371361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.380614 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.380591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7mt\" (UniqueName: \"kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.875332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.875303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:23.877716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:23.877691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") pod \"maas-api-e2e-migrate-cdca1605-976bcb87-7st97\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:24.069478 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:24.069443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:24.235991 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:24.235951 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:27:24.241100 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:27:24.241069 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc054f45f_4bd4_4f44_a425_766926463348.slice/crio-a35a4b88c75eba828557867eec8bbd305be264b946bd2898c6387729c387de68 WatchSource:0}: Error finding container a35a4b88c75eba828557867eec8bbd305be264b946bd2898c6387729c387de68: Status 404 returned error can't find the container with id a35a4b88c75eba828557867eec8bbd305be264b946bd2898c6387729c387de68 Jul 15 08:27:25.044016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:25.043934 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" event={"ID":"c054f45f-4bd4-4f44-a425-766926463348","Type":"ContainerStarted","Data":"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c"} Jul 15 08:27:25.044016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:25.043992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" event={"ID":"c054f45f-4bd4-4f44-a425-766926463348","Type":"ContainerStarted","Data":"a35a4b88c75eba828557867eec8bbd305be264b946bd2898c6387729c387de68"} Jul 15 08:27:25.044248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:25.044061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:25.062273 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:25.062226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" podStartSLOduration=1.5531291170000001 podStartE2EDuration="2.062213815s" totalImagesPullingTime="509.084698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:27:25.06017696 +0000 UTC m=+1622.479921787" watchObservedRunningTime="2026-07-15 08:27:25.062213815 +0000 UTC m=+1622.481958618" Jul 15 08:27:27.621342 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.621307 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:27.625075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.625058 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:27.627884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.627862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:27:27.632131 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.632109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:27.710782 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.710752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-z482h\" (UID: \"60a09ea1-6640-44d7-bae0-f3b852a3f1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:27.811546 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.811522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-z482h\" (UID: \"60a09ea1-6640-44d7-bae0-f3b852a3f1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:27.812180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.812161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-z482h\" (UID: \"60a09ea1-6640-44d7-bae0-f3b852a3f1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:27.937775 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:27.937712 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:28.069735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:28.069713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:28.072236 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:27:28.072208 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a09ea1_6640_44d7_bae0_f3b852a3f1f1.slice/crio-f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a WatchSource:0}: Error finding container f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a: Status 404 returned error can't find the container with id f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a Jul 15 08:27:29.069459 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:29.069350 2565 generic.go:356] "Generic (PLEG): container finished" podID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerID="056b143f7efcf6a6423f71beee7d7c325ae111207aeec4ffc08f6a3f8e2f673c" exitCode=7 Jul 15 08:27:29.069459 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:29.069431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" event={"ID":"60a09ea1-6640-44d7-bae0-f3b852a3f1f1","Type":"ContainerDied","Data":"056b143f7efcf6a6423f71beee7d7c325ae111207aeec4ffc08f6a3f8e2f673c"} Jul 15 08:27:29.069837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:29.069481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" event={"ID":"60a09ea1-6640-44d7-bae0-f3b852a3f1f1","Type":"ContainerStarted","Data":"f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a"} Jul 15 08:27:29.069837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:29.069611 2565 scope.go:117] "RemoveContainer" containerID="056b143f7efcf6a6423f71beee7d7c325ae111207aeec4ffc08f6a3f8e2f673c" Jul 15 08:27:30.076351 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:30.076311 2565 generic.go:356] "Generic (PLEG): container finished" podID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerID="773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b" exitCode=7 Jul 15 08:27:30.076820 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:30.076414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" event={"ID":"60a09ea1-6640-44d7-bae0-f3b852a3f1f1","Type":"ContainerDied","Data":"773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b"} Jul 15 08:27:30.076820 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:30.076493 2565 scope.go:117] "RemoveContainer" containerID="056b143f7efcf6a6423f71beee7d7c325ae111207aeec4ffc08f6a3f8e2f673c" Jul 15 08:27:30.076820 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:30.076670 2565 scope.go:117] "RemoveContainer" containerID="773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b" Jul 15 08:27:30.076943 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:27:30.076881 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cdca1605-z482h_odh-ai-gateway-infra(60a09ea1-6640-44d7-bae0-f3b852a3f1f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" Jul 15 08:27:31.054298 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:31.054265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:27:31.084533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:31.084506 2565 scope.go:117] "RemoveContainer" containerID="773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b" Jul 15 08:27:31.084977 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:27:31.084746 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cdca1605-z482h_odh-ai-gateway-infra(60a09ea1-6640-44d7-bae0-f3b852a3f1f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" Jul 15 08:27:46.111382 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:46.111349 2565 scope.go:117] "RemoveContainer" containerID="773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b" Jul 15 08:27:47.168469 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:47.168427 2565 generic.go:356] "Generic (PLEG): container finished" podID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerID="0499dcce7b0685d9084709d05e66a315cdf9cf018b0b846c388660e2181bb304" exitCode=0 Jul 15 08:27:47.168888 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:47.168442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" event={"ID":"60a09ea1-6640-44d7-bae0-f3b852a3f1f1","Type":"ContainerDied","Data":"0499dcce7b0685d9084709d05e66a315cdf9cf018b0b846c388660e2181bb304"} Jul 15 08:27:47.168888 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:47.168575 2565 scope.go:117] "RemoveContainer" containerID="773224851a95e1fd0511c05e56d79848dbb4ffe88f3a81f4df762f8fc48b3b7b" Jul 15 08:27:48.200866 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:48.200825 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:48.313626 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:48.313600 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:48.402511 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:48.402482 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca\") pod \"60a09ea1-6640-44d7-bae0-f3b852a3f1f1\" (UID: \"60a09ea1-6640-44d7-bae0-f3b852a3f1f1\") " Jul 15 08:27:48.402859 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:48.402837 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60a09ea1-6640-44d7-bae0-f3b852a3f1f1" (UID: "60a09ea1-6640-44d7-bae0-f3b852a3f1f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:27:48.504132 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:48.504104 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a09ea1-6640-44d7-bae0-f3b852a3f1f1-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:27:49.183226 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:49.183192 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" Jul 15 08:27:49.183226 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:49.183202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h" event={"ID":"60a09ea1-6640-44d7-bae0-f3b852a3f1f1","Type":"ContainerDied","Data":"f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a"} Jul 15 08:27:49.183465 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:49.183253 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f2688e678a29de2eb2bc307361bfdd2ee8c0bb6c145487acb58e2d3d047f8a" Jul 15 08:27:49.214177 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:49.214141 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:49.226314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:49.220424 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-z482h"] Jul 15 08:27:51.117129 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:27:51.117093 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" path="/var/lib/kubelet/pods/60a09ea1-6640-44d7-bae0-f3b852a3f1f1/volumes" Jul 15 08:28:27.671278 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671239 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq"] Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671667 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671678 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671688 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671696 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671708 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671713 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671927 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671781 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671927 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671789 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.671927 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.671799 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a09ea1-6640-44d7-bae0-f3b852a3f1f1" containerName="revoke-keys" Jul 15 08:28:27.675037 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.675020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:27.677769 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.677747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:28:27.684139 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.684111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq"] Jul 15 08:28:27.785655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.785617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq\" (UID: \"07e1d1a8-909c-4d3d-9e47-957385d10f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:27.887088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.887050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq\" (UID: \"07e1d1a8-909c-4d3d-9e47-957385d10f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:27.887914 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.887883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq\" (UID: \"07e1d1a8-909c-4d3d-9e47-957385d10f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:27.987620 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:27.987581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:28.129950 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:28.129914 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq"] Jul 15 08:28:28.132132 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:28:28.132106 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e1d1a8_909c_4d3d_9e47_957385d10f4b.slice/crio-68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c WatchSource:0}: Error finding container 68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c: Status 404 returned error can't find the container with id 68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c Jul 15 08:28:28.389273 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:28.389245 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" event={"ID":"07e1d1a8-909c-4d3d-9e47-957385d10f4b","Type":"ContainerStarted","Data":"68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c"} Jul 15 08:28:29.396028 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:29.395993 2565 generic.go:356] "Generic (PLEG): container finished" podID="07e1d1a8-909c-4d3d-9e47-957385d10f4b" containerID="85470050a72fa475568ba0ef4f2b66c902b0c52d265f52035e53f049ef7ea0bf" exitCode=0 Jul 15 08:28:29.396461 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:29.396082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" event={"ID":"07e1d1a8-909c-4d3d-9e47-957385d10f4b","Type":"ContainerDied","Data":"85470050a72fa475568ba0ef4f2b66c902b0c52d265f52035e53f049ef7ea0bf"} Jul 15 08:28:30.540044 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:30.540014 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:30.615008 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:30.614974 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca\") pod \"07e1d1a8-909c-4d3d-9e47-957385d10f4b\" (UID: \"07e1d1a8-909c-4d3d-9e47-957385d10f4b\") " Jul 15 08:28:30.615352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:30.615325 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07e1d1a8-909c-4d3d-9e47-957385d10f4b" (UID: "07e1d1a8-909c-4d3d-9e47-957385d10f4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:28:30.716513 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:30.716393 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e1d1a8-909c-4d3d-9e47-957385d10f4b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:28:31.408024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:31.407941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" Jul 15 08:28:31.408180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:31.407940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq" event={"ID":"07e1d1a8-909c-4d3d-9e47-957385d10f4b","Type":"ContainerDied","Data":"68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c"} Jul 15 08:28:31.408180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:31.408067 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68aada0c9327131b1166afe6508de0912b57bf04359af1546df3a314579f814c" Jul 15 08:28:33.514747 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.514719 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:28:33.515162 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.514896 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" podUID="c054f45f-4bd4-4f44-a425-766926463348" containerName="maas-api" containerID="cri-o://f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c" gracePeriod=30 Jul 15 08:28:33.766689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.766625 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:28:33.845625 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.845600 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w7mt\" (UniqueName: \"kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt\") pod \"c054f45f-4bd4-4f44-a425-766926463348\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " Jul 15 08:28:33.845771 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.845635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") pod \"c054f45f-4bd4-4f44-a425-766926463348\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " Jul 15 08:28:33.845771 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.845684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca\") pod \"c054f45f-4bd4-4f44-a425-766926463348\" (UID: \"c054f45f-4bd4-4f44-a425-766926463348\") " Jul 15 08:28:33.846088 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.846061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c054f45f-4bd4-4f44-a425-766926463348" (UID: "c054f45f-4bd4-4f44-a425-766926463348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:28:33.847673 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.847644 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt" (OuterVolumeSpecName: "kube-api-access-6w7mt") pod "c054f45f-4bd4-4f44-a425-766926463348" (UID: "c054f45f-4bd4-4f44-a425-766926463348"). InnerVolumeSpecName "kube-api-access-6w7mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:28:33.847772 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.847677 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c054f45f-4bd4-4f44-a425-766926463348" (UID: "c054f45f-4bd4-4f44-a425-766926463348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:28:33.947072 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.947048 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7mt\" (UniqueName: \"kubernetes.io/projected/c054f45f-4bd4-4f44-a425-766926463348-kube-api-access-6w7mt\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:28:33.947072 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.947070 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c054f45f-4bd4-4f44-a425-766926463348-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:28:33.947207 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:33.947080 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c054f45f-4bd4-4f44-a425-766926463348-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:28:34.425689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.425656 2565 generic.go:356] "Generic (PLEG): container finished" podID="c054f45f-4bd4-4f44-a425-766926463348" containerID="f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c" exitCode=0 Jul 15 08:28:34.425865 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.425698 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" Jul 15 08:28:34.425865 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.425743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" event={"ID":"c054f45f-4bd4-4f44-a425-766926463348","Type":"ContainerDied","Data":"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c"} Jul 15 08:28:34.425865 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.425796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97" event={"ID":"c054f45f-4bd4-4f44-a425-766926463348","Type":"ContainerDied","Data":"a35a4b88c75eba828557867eec8bbd305be264b946bd2898c6387729c387de68"} Jul 15 08:28:34.425865 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.425820 2565 scope.go:117] "RemoveContainer" containerID="f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c" Jul 15 08:28:34.435222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.435198 2565 scope.go:117] "RemoveContainer" containerID="f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c" Jul 15 08:28:34.435523 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:28:34.435502 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c\": container with ID starting with f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c not found: ID does not exist" containerID="f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c" Jul 15 08:28:34.435590 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.435530 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c"} err="failed to get container status \"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c\": rpc error: code = NotFound desc = could not find container \"f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c\": container with ID starting with f8f841d928c80107ae7113532ee268ca4038d86e8e171e5951246aa4696d668c not found: ID does not exist" Jul 15 08:28:34.453162 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.453086 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:28:34.460963 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:34.460940 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdca1605-976bcb87-7st97"] Jul 15 08:28:35.123685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:35.123644 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c054f45f-4bd4-4f44-a425-766926463348" path="/var/lib/kubelet/pods/c054f45f-4bd4-4f44-a425-766926463348/volumes" Jul 15 08:28:42.608493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.608457 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:28:42.608966 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.608950 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e1d1a8-909c-4d3d-9e47-957385d10f4b" containerName="revoke-keys" Jul 15 08:28:42.609007 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.608969 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e1d1a8-909c-4d3d-9e47-957385d10f4b" containerName="revoke-keys" Jul 15 08:28:42.609007 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.608985 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c054f45f-4bd4-4f44-a425-766926463348" containerName="maas-api" Jul 15 08:28:42.609007 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.608990 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c054f45f-4bd4-4f44-a425-766926463348" containerName="maas-api" Jul 15 08:28:42.609101 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.609065 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e1d1a8-909c-4d3d-9e47-957385d10f4b" containerName="revoke-keys" Jul 15 08:28:42.609101 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.609079 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c054f45f-4bd4-4f44-a425-766926463348" containerName="maas-api" Jul 15 08:28:42.613711 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.613688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.616155 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.616131 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4dbdead5-serving-cert\"" Jul 15 08:28:42.623880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.623854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:28:42.733501 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.733456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.733687 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.733530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknsc\" (UniqueName: \"kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.733687 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.733666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.834878 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.834842 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.835058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.834893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.835058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.834931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknsc\" (UniqueName: \"kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.835147 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:28:42.835049 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-serving-cert: secret "maas-api-e2e-ait-4dbdead5-serving-cert" not found Jul 15 08:28:42.835147 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:28:42.835126 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls podName:997a3477-ab4e-4572-a952-bd1eeca70de0 nodeName:}" failed. No retries permitted until 2026-07-15 08:28:43.335109379 +0000 UTC m=+1700.754854160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls") pod "maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" (UID: "997a3477-ab4e-4572-a952-bd1eeca70de0") : secret "maas-api-e2e-ait-4dbdead5-serving-cert" not found Jul 15 08:28:42.835542 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.835523 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:42.845033 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:42.845004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknsc\" (UniqueName: \"kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:43.340222 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:43.340172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:43.342851 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:43.342821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") pod \"maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:43.524700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:43.524667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:43.680660 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:43.680635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:28:43.682005 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:28:43.681977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997a3477_ab4e_4572_a952_bd1eeca70de0.slice/crio-484b708ceaec284011d20db69df5cc8fb9b13d9d00611eede7e36690bb012447 WatchSource:0}: Error finding container 484b708ceaec284011d20db69df5cc8fb9b13d9d00611eede7e36690bb012447: Status 404 returned error can't find the container with id 484b708ceaec284011d20db69df5cc8fb9b13d9d00611eede7e36690bb012447 Jul 15 08:28:44.483237 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:44.483202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" event={"ID":"997a3477-ab4e-4572-a952-bd1eeca70de0","Type":"ContainerStarted","Data":"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1"} Jul 15 08:28:44.483455 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:44.483259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" event={"ID":"997a3477-ab4e-4572-a952-bd1eeca70de0","Type":"ContainerStarted","Data":"484b708ceaec284011d20db69df5cc8fb9b13d9d00611eede7e36690bb012447"} Jul 15 08:28:44.483455 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:44.483322 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:44.509192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:44.509139 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" podStartSLOduration=2.064328296 podStartE2EDuration="2.509122524s" totalImagesPullingTime="444.794228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:28:44.506384589 +0000 UTC m=+1701.926129392" watchObservedRunningTime="2026-07-15 08:28:44.509122524 +0000 UTC m=+1701.928867327" Jul 15 08:28:47.608476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.608437 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:28:47.612534 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.612501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:28:47.617821 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.617800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:28:47.629716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.629690 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:28:47.690056 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.690006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg\" (UID: \"4c386054-d750-42dd-8a50-92d12734e914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:28:47.790626 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.790584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg\" (UID: \"4c386054-d750-42dd-8a50-92d12734e914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:28:47.791235 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.791211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg\" (UID: \"4c386054-d750-42dd-8a50-92d12734e914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:28:47.923573 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:47.923472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:28:48.075629 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:48.075599 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:28:48.077396 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:28:48.077370 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c386054_d750_42dd_8a50_92d12734e914.slice/crio-c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc WatchSource:0}: Error finding container c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc: Status 404 returned error can't find the container with id c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc Jul 15 08:28:48.506925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:48.506887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerStarted","Data":"b1117be836b6e1a0e53b3c094766c8f07dcf8b732eedc5b9d7c6e83ac169a8ed"} Jul 15 08:28:48.507084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:48.506940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerStarted","Data":"c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc"} Jul 15 08:28:48.543471 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:48.543348 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" podStartSLOduration=1.543328772 podStartE2EDuration="1.543328772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:28:48.539734288 +0000 UTC m=+1705.959479092" watchObservedRunningTime="2026-07-15 08:28:48.543328772 +0000 UTC m=+1705.963073576" Jul 15 08:28:49.515147 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:49.515113 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c386054-d750-42dd-8a50-92d12734e914" containerID="b1117be836b6e1a0e53b3c094766c8f07dcf8b732eedc5b9d7c6e83ac169a8ed" exitCode=7 Jul 15 08:28:49.515588 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:49.515198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerDied","Data":"b1117be836b6e1a0e53b3c094766c8f07dcf8b732eedc5b9d7c6e83ac169a8ed"} Jul 15 08:28:49.515588 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:49.515356 2565 scope.go:117] "RemoveContainer" containerID="b1117be836b6e1a0e53b3c094766c8f07dcf8b732eedc5b9d7c6e83ac169a8ed" Jul 15 08:28:50.494672 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:50.494647 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:28:50.522562 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:50.522518 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c386054-d750-42dd-8a50-92d12734e914" containerID="55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7" exitCode=7 Jul 15 08:28:50.522996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:50.522561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerDied","Data":"55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7"} Jul 15 08:28:50.522996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:50.522646 2565 scope.go:117] "RemoveContainer" containerID="b1117be836b6e1a0e53b3c094766c8f07dcf8b732eedc5b9d7c6e83ac169a8ed" Jul 15 08:28:50.522996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:50.522805 2565 scope.go:117] "RemoveContainer" containerID="55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7" Jul 15 08:28:50.523149 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:28:50.523070 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg_odh-ai-gateway-infra(4c386054-d750-42dd-8a50-92d12734e914)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" podUID="4c386054-d750-42dd-8a50-92d12734e914" Jul 15 08:28:51.529929 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:28:51.529899 2565 scope.go:117] "RemoveContainer" containerID="55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7" Jul 15 08:28:51.530333 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:28:51.530102 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg_odh-ai-gateway-infra(4c386054-d750-42dd-8a50-92d12734e914)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" podUID="4c386054-d750-42dd-8a50-92d12734e914" Jul 15 08:29:05.110579 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:05.110547 2565 scope.go:117] "RemoveContainer" containerID="55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7" Jul 15 08:29:05.607844 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:05.607808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerStarted","Data":"a30574635e07584951d7c0b1f31f64902e5afba6a800e98aca78de5576d177e1"} Jul 15 08:29:06.614912 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:06.614875 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c386054-d750-42dd-8a50-92d12734e914" containerID="a30574635e07584951d7c0b1f31f64902e5afba6a800e98aca78de5576d177e1" exitCode=0 Jul 15 08:29:06.615347 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:06.614953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerDied","Data":"a30574635e07584951d7c0b1f31f64902e5afba6a800e98aca78de5576d177e1"} Jul 15 08:29:06.615347 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:06.615026 2565 scope.go:117] "RemoveContainer" containerID="55787f0cb0daa2f948f8a27f2b7f1abd65841bf19be39641f16be5a4bf8b65c7" Jul 15 08:29:06.641696 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:06.641664 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:29:07.762945 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:07.762918 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:29:07.789842 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:07.789813 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca\") pod \"4c386054-d750-42dd-8a50-92d12734e914\" (UID: \"4c386054-d750-42dd-8a50-92d12734e914\") " Jul 15 08:29:07.790171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:07.790150 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c386054-d750-42dd-8a50-92d12734e914" (UID: "4c386054-d750-42dd-8a50-92d12734e914"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:29:07.890613 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:07.890504 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c386054-d750-42dd-8a50-92d12734e914-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:29:08.629436 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:08.629379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" event={"ID":"4c386054-d750-42dd-8a50-92d12734e914","Type":"ContainerDied","Data":"c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc"} Jul 15 08:29:08.629640 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:08.629461 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e630779e754aa6a5dea88bcfb0d6bded8aedadde6004b79eac7104054215dc" Jul 15 08:29:08.629640 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:08.629396 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg" Jul 15 08:29:08.661682 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:08.661650 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:29:08.668546 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:08.668514 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-lwfqg"] Jul 15 08:29:09.117323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:09.117282 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c386054-d750-42dd-8a50-92d12734e914" path="/var/lib/kubelet/pods/4c386054-d750-42dd-8a50-92d12734e914/volumes" Jul 15 08:29:47.668587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.668542 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k"] Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.668953 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.668965 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.668981 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.668987 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.669059 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.669209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.669067 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:29:47.672569 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.672546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:47.675085 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.675058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:29:47.679467 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.679384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k"] Jul 15 08:29:47.791098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.791064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k\" (UID: \"461bb60b-2fef-4140-919e-4d75a09304e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:47.892223 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.892187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k\" (UID: \"461bb60b-2fef-4140-919e-4d75a09304e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:47.893435 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.893382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k\" (UID: \"461bb60b-2fef-4140-919e-4d75a09304e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:47.987385 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:47.987351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:48.122370 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:48.122343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k"] Jul 15 08:29:48.124161 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:29:48.124134 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461bb60b_2fef_4140_919e_4d75a09304e2.slice/crio-96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b WatchSource:0}: Error finding container 96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b: Status 404 returned error can't find the container with id 96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b Jul 15 08:29:48.842284 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:48.842251 2565 generic.go:356] "Generic (PLEG): container finished" podID="461bb60b-2fef-4140-919e-4d75a09304e2" containerID="6ceb24ca658af35faee8e4d174b513b1ff0520291b0e53a749cd30ff4c6e2dae" exitCode=0 Jul 15 08:29:48.842717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:48.842337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" event={"ID":"461bb60b-2fef-4140-919e-4d75a09304e2","Type":"ContainerDied","Data":"6ceb24ca658af35faee8e4d174b513b1ff0520291b0e53a749cd30ff4c6e2dae"} Jul 15 08:29:48.842717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:48.842423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" event={"ID":"461bb60b-2fef-4140-919e-4d75a09304e2","Type":"ContainerStarted","Data":"96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b"} Jul 15 08:29:48.913975 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:29:48.913916 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461bb60b_2fef_4140_919e_4d75a09304e2.slice/crio-conmon-6ceb24ca658af35faee8e4d174b513b1ff0520291b0e53a749cd30ff4c6e2dae.scope\": RecentStats: unable to find data in memory cache]" Jul 15 08:29:49.988047 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:49.988023 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:50.114480 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.114358 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca\") pod \"461bb60b-2fef-4140-919e-4d75a09304e2\" (UID: \"461bb60b-2fef-4140-919e-4d75a09304e2\") " Jul 15 08:29:50.114729 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.114706 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "461bb60b-2fef-4140-919e-4d75a09304e2" (UID: "461bb60b-2fef-4140-919e-4d75a09304e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:29:50.216085 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.216043 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461bb60b-2fef-4140-919e-4d75a09304e2-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:29:50.853322 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.853283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" event={"ID":"461bb60b-2fef-4140-919e-4d75a09304e2","Type":"ContainerDied","Data":"96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b"} Jul 15 08:29:50.853515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.853332 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96da785433bf62c49f813505dbfd88fbf73f8aaa9b3d83c5fc467bc57f50325b" Jul 15 08:29:50.853515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:50.853299 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k" Jul 15 08:29:58.221422 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.221375 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:29:58.221874 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.221617 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" podUID="997a3477-ab4e-4572-a952-bd1eeca70de0" containerName="maas-api" containerID="cri-o://dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1" gracePeriod=30 Jul 15 08:29:58.470021 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.469997 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:29:58.598925 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.598895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qknsc\" (UniqueName: \"kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc\") pod \"997a3477-ab4e-4572-a952-bd1eeca70de0\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " Jul 15 08:29:58.599093 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.598939 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca\") pod \"997a3477-ab4e-4572-a952-bd1eeca70de0\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " Jul 15 08:29:58.599093 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.599060 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") pod \"997a3477-ab4e-4572-a952-bd1eeca70de0\" (UID: \"997a3477-ab4e-4572-a952-bd1eeca70de0\") " Jul 15 08:29:58.599297 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.599271 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "997a3477-ab4e-4572-a952-bd1eeca70de0" (UID: "997a3477-ab4e-4572-a952-bd1eeca70de0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:29:58.599475 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.599458 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997a3477-ab4e-4572-a952-bd1eeca70de0-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:29:58.600787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.600766 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc" (OuterVolumeSpecName: "kube-api-access-qknsc") pod "997a3477-ab4e-4572-a952-bd1eeca70de0" (UID: "997a3477-ab4e-4572-a952-bd1eeca70de0"). InnerVolumeSpecName "kube-api-access-qknsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:29:58.600903 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.600882 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "997a3477-ab4e-4572-a952-bd1eeca70de0" (UID: "997a3477-ab4e-4572-a952-bd1eeca70de0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:29:58.700472 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.700432 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknsc\" (UniqueName: \"kubernetes.io/projected/997a3477-ab4e-4572-a952-bd1eeca70de0-kube-api-access-qknsc\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:29:58.700472 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.700465 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997a3477-ab4e-4572-a952-bd1eeca70de0-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:29:58.898447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.898336 2565 generic.go:356] "Generic (PLEG): container finished" podID="997a3477-ab4e-4572-a952-bd1eeca70de0" containerID="dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1" exitCode=0 Jul 15 08:29:58.898447 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.898421 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" Jul 15 08:29:58.898627 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.898435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" event={"ID":"997a3477-ab4e-4572-a952-bd1eeca70de0","Type":"ContainerDied","Data":"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1"} Jul 15 08:29:58.898627 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.898518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj" event={"ID":"997a3477-ab4e-4572-a952-bd1eeca70de0","Type":"ContainerDied","Data":"484b708ceaec284011d20db69df5cc8fb9b13d9d00611eede7e36690bb012447"} Jul 15 08:29:58.898627 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.898542 2565 scope.go:117] "RemoveContainer" containerID="dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1" Jul 15 08:29:58.909365 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.909342 2565 scope.go:117] "RemoveContainer" containerID="dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1" Jul 15 08:29:58.909694 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:29:58.909672 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1\": container with ID starting with dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1 not found: ID does not exist" containerID="dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1" Jul 15 08:29:58.909767 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.909701 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1"} err="failed to get container status \"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1\": rpc error: code = NotFound desc = could not find container \"dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1\": container with ID starting with dbf5bdecb7d978b992ef94fad26f5279fc9a9688095002b19e9caffcadcb17d1 not found: ID does not exist" Jul 15 08:29:58.920101 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.920066 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:29:58.924205 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:58.924180 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dbdead5-7bbbfff758-pzvkj"] Jul 15 08:29:59.117004 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:29:59.116969 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997a3477-ab4e-4572-a952-bd1eeca70de0" path="/var/lib/kubelet/pods/997a3477-ab4e-4572-a952-bd1eeca70de0/volumes" Jul 15 08:30:00.139070 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139032 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8"] Jul 15 08:30:00.139506 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139492 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997a3477-ab4e-4572-a952-bd1eeca70de0" containerName="maas-api" Jul 15 08:30:00.139557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139509 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="997a3477-ab4e-4572-a952-bd1eeca70de0" containerName="maas-api" Jul 15 08:30:00.139557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139522 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:30:00.139557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139527 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:30:00.139557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139536 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461bb60b-2fef-4140-919e-4d75a09304e2" containerName="revoke-keys" Jul 15 08:30:00.139557 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139542 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="461bb60b-2fef-4140-919e-4d75a09304e2" containerName="revoke-keys" Jul 15 08:30:00.139709 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139636 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="997a3477-ab4e-4572-a952-bd1eeca70de0" containerName="maas-api" Jul 15 08:30:00.139709 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139648 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c386054-d750-42dd-8a50-92d12734e914" containerName="revoke-keys" Jul 15 08:30:00.139709 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.139657 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="461bb60b-2fef-4140-919e-4d75a09304e2" containerName="revoke-keys" Jul 15 08:30:00.144217 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.144199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" Jul 15 08:30:00.146935 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.146900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:30:00.149164 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.149137 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8"] Jul 15 08:30:00.155664 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.155642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" Jul 15 08:30:00.315993 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.315962 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8"] Jul 15 08:30:00.317676 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:30:00.317637 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209af120_e927_42aa_8e1e_74fddae0c24d.slice/crio-efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15 WatchSource:0}: Error finding container efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15: Status 404 returned error can't find the container with id efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15 Jul 15 08:30:00.911787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.911698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" event={"ID":"209af120-e927-42aa-8e1e-74fddae0c24d","Type":"ContainerStarted","Data":"81062fb001c193849faba0f6b0c89317b1be466e62ffaf013d17720f95a25dbf"} Jul 15 08:30:00.911787 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.911752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" event={"ID":"209af120-e927-42aa-8e1e-74fddae0c24d","Type":"ContainerStarted","Data":"efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15"} Jul 15 08:30:00.927516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:00.927469 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" podStartSLOduration=0.927455237 podStartE2EDuration="927.455237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:30:00.926190345 +0000 UTC m=+1778.345935149" watchObservedRunningTime="2026-07-15 08:30:00.927455237 +0000 UTC m=+1778.347200039" Jul 15 08:30:01.918678 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:01.918645 2565 generic.go:356] "Generic (PLEG): container finished" podID="209af120-e927-42aa-8e1e-74fddae0c24d" containerID="81062fb001c193849faba0f6b0c89317b1be466e62ffaf013d17720f95a25dbf" exitCode=0 Jul 15 08:30:01.919095 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:01.918738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" event={"ID":"209af120-e927-42aa-8e1e-74fddae0c24d","Type":"ContainerDied","Data":"81062fb001c193849faba0f6b0c89317b1be466e62ffaf013d17720f95a25dbf"} Jul 15 08:30:03.059989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:03.059961 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" Jul 15 08:30:03.931334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:03.931300 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" Jul 15 08:30:03.931334 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:03.931315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735070-nw2c8" event={"ID":"209af120-e927-42aa-8e1e-74fddae0c24d","Type":"ContainerDied","Data":"efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15"} Jul 15 08:30:03.931577 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:03.931389 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efb11858bb3e039cbe02926981be5c78b72b3daf5497ddcf8f3227b442506a15" Jul 15 08:30:15.397023 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.396929 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:30:15.397587 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.397550 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209af120-e927-42aa-8e1e-74fddae0c24d" containerName="cleanup" Jul 15 08:30:15.397667 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.397590 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="209af120-e927-42aa-8e1e-74fddae0c24d" containerName="cleanup" Jul 15 08:30:15.397741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.397727 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="209af120-e927-42aa-8e1e-74fddae0c24d" containerName="cleanup" Jul 15 08:30:15.401053 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.401032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.403725 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.403700 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93b7ed10-serving-cert\"" Jul 15 08:30:15.419058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.419030 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:30:15.463754 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.463717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7w5x\" (UniqueName: \"kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.463938 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.463825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.463938 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.463876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.565204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.565164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.565445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.565218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.565445 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.565273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7w5x\" (UniqueName: \"kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.565934 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.565887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.567733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.567713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.583519 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.583485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7w5x\" (UniqueName: \"kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x\") pod \"maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.711754 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.711655 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:15.885763 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.885731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:30:15.887676 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:30:15.887648 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode964cdca_f5ec_4fd0_94b8_ad622e44d93f.slice/crio-359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de WatchSource:0}: Error finding container 359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de: Status 404 returned error can't find the container with id 359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de Jul 15 08:30:15.999743 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:15.999698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" event={"ID":"e964cdca-f5ec-4fd0-94b8-ad622e44d93f","Type":"ContainerStarted","Data":"359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de"} Jul 15 08:30:17.007770 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:17.007733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" event={"ID":"e964cdca-f5ec-4fd0-94b8-ad622e44d93f","Type":"ContainerStarted","Data":"0febbcf74e46c8905afbd4582ee8a9cbba48a62e52cd0479ffc32949c0badfc6"} Jul 15 08:30:17.008184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:17.007869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:17.050781 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:17.050730 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" podStartSLOduration=1.635842447 podStartE2EDuration="2.050714724s" totalImagesPullingTime="414.872277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:30:17.050206584 +0000 UTC m=+1794.469951415" watchObservedRunningTime="2026-07-15 08:30:17.050714724 +0000 UTC m=+1794.470459577" Jul 15 08:30:19.702332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.702293 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:19.706021 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.705995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:19.708384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.708363 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:30:19.714197 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.714112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:19.805716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.805678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps\" (UID: \"df9d3a45-b59e-4c37-9b5e-04a17c185c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:19.906919 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.906880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps\" (UID: \"df9d3a45-b59e-4c37-9b5e-04a17c185c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:19.907536 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:19.907516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps\" (UID: \"df9d3a45-b59e-4c37-9b5e-04a17c185c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:20.019130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:20.019098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:20.159141 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:20.159114 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:20.160977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:30:20.160945 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9d3a45_b59e_4c37_9b5e_04a17c185c51.slice/crio-94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b WatchSource:0}: Error finding container 94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b: Status 404 returned error can't find the container with id 94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b Jul 15 08:30:21.033974 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:21.033602 2565 generic.go:356] "Generic (PLEG): container finished" podID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerID="1004e864c7f93b53abe39ec934548c49195139accc1aff3a95571204fa6a108b" exitCode=7 Jul 15 08:30:21.033974 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:21.033672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" event={"ID":"df9d3a45-b59e-4c37-9b5e-04a17c185c51","Type":"ContainerDied","Data":"1004e864c7f93b53abe39ec934548c49195139accc1aff3a95571204fa6a108b"} Jul 15 08:30:21.033974 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:21.033727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" event={"ID":"df9d3a45-b59e-4c37-9b5e-04a17c185c51","Type":"ContainerStarted","Data":"94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b"} Jul 15 08:30:21.034564 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:21.034070 2565 scope.go:117] "RemoveContainer" containerID="1004e864c7f93b53abe39ec934548c49195139accc1aff3a95571204fa6a108b" Jul 15 08:30:22.040794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:22.040753 2565 generic.go:356] "Generic (PLEG): container finished" podID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerID="5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66" exitCode=7 Jul 15 08:30:22.041187 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:22.040837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" event={"ID":"df9d3a45-b59e-4c37-9b5e-04a17c185c51","Type":"ContainerDied","Data":"5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66"} Jul 15 08:30:22.041187 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:22.040902 2565 scope.go:117] "RemoveContainer" containerID="1004e864c7f93b53abe39ec934548c49195139accc1aff3a95571204fa6a108b" Jul 15 08:30:22.041187 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:22.040994 2565 scope.go:117] "RemoveContainer" containerID="5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66" Jul 15 08:30:22.041328 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:30:22.041229 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps_odh-ai-gateway-infra(df9d3a45-b59e-4c37-9b5e-04a17c185c51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" Jul 15 08:30:23.018930 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.018900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:30:23.048310 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.048284 2565 scope.go:117] "RemoveContainer" containerID="5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66" Jul 15 08:30:23.048718 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:30:23.048499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps_odh-ai-gateway-infra(df9d3a45-b59e-4c37-9b5e-04a17c185c51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" Jul 15 08:30:23.210308 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.210196 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:30:23.229716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.214995 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:30:23.229716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.218062 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:30:23.229716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:23.222332 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:30:37.111342 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:37.111305 2565 scope.go:117] "RemoveContainer" containerID="5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66" Jul 15 08:30:38.134193 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:38.134104 2565 generic.go:356] "Generic (PLEG): container finished" podID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerID="c2aaa3f3d2d276d606bda2ab4aef78295662fca1851c3a201a55f6f0573ec353" exitCode=0 Jul 15 08:30:38.134193 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:38.134171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" event={"ID":"df9d3a45-b59e-4c37-9b5e-04a17c185c51","Type":"ContainerDied","Data":"c2aaa3f3d2d276d606bda2ab4aef78295662fca1851c3a201a55f6f0573ec353"} Jul 15 08:30:38.134634 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:38.134227 2565 scope.go:117] "RemoveContainer" containerID="5635a08b6d513d334f16b5e741fce1e15df89fd949e081528fbafbedb2cb1d66" Jul 15 08:30:39.175670 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:39.175634 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:39.285367 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:39.285337 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:39.408790 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:39.408705 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca\") pod \"df9d3a45-b59e-4c37-9b5e-04a17c185c51\" (UID: \"df9d3a45-b59e-4c37-9b5e-04a17c185c51\") " Jul 15 08:30:39.409076 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:39.409054 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df9d3a45-b59e-4c37-9b5e-04a17c185c51" (UID: "df9d3a45-b59e-4c37-9b5e-04a17c185c51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:30:39.509387 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:39.509356 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3a45-b59e-4c37-9b5e-04a17c185c51-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:30:40.147462 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:40.147432 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" Jul 15 08:30:40.147616 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:40.147428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps" event={"ID":"df9d3a45-b59e-4c37-9b5e-04a17c185c51","Type":"ContainerDied","Data":"94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b"} Jul 15 08:30:40.147616 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:40.147552 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94551039cf193bcf3d487a1f9b36c44e290a0bfe74bf67ddec924ffc01197c6b" Jul 15 08:30:40.179599 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:40.177737 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:40.184858 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:40.184827 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-gs7ps"] Jul 15 08:30:41.116837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:30:41.116805 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" path="/var/lib/kubelet/pods/df9d3a45-b59e-4c37-9b5e-04a17c185c51/volumes" Jul 15 08:31:19.747727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.747691 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5"] Jul 15 08:31:19.748192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748120 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.748192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748131 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.748192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748149 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.748192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748155 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.748330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748227 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.748330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.748237 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:31:19.751702 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.751685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:19.754058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.754032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:31:19.758912 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.758784 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5"] Jul 15 08:31:19.801678 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.801641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-279x5\" (UID: \"2c21d8d9-79e8-47d5-b135-556be5addd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:19.902795 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.902759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-279x5\" (UID: \"2c21d8d9-79e8-47d5-b135-556be5addd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:19.903385 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:19.903367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93b7ed10-279x5\" (UID: \"2c21d8d9-79e8-47d5-b135-556be5addd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:20.064552 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:20.064450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:20.202025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:20.201985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5"] Jul 15 08:31:20.205977 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:31:20.205951 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c21d8d9_79e8_47d5_b135_556be5addd9e.slice/crio-254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a WatchSource:0}: Error finding container 254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a: Status 404 returned error can't find the container with id 254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a Jul 15 08:31:20.364638 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:20.364552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" event={"ID":"2c21d8d9-79e8-47d5-b135-556be5addd9e","Type":"ContainerStarted","Data":"254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a"} Jul 15 08:31:21.371782 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:21.371747 2565 generic.go:356] "Generic (PLEG): container finished" podID="2c21d8d9-79e8-47d5-b135-556be5addd9e" containerID="0ec9cd661798daa95d42a3fed6d6f3d941b35783ff552c2cd6733391ffb97c59" exitCode=0 Jul 15 08:31:21.372167 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:21.371816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" event={"ID":"2c21d8d9-79e8-47d5-b135-556be5addd9e","Type":"ContainerDied","Data":"0ec9cd661798daa95d42a3fed6d6f3d941b35783ff552c2cd6733391ffb97c59"} Jul 15 08:31:22.512319 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:22.512295 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:22.523417 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:22.523381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca\") pod \"2c21d8d9-79e8-47d5-b135-556be5addd9e\" (UID: \"2c21d8d9-79e8-47d5-b135-556be5addd9e\") " Jul 15 08:31:22.523692 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:22.523670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c21d8d9-79e8-47d5-b135-556be5addd9e" (UID: "2c21d8d9-79e8-47d5-b135-556be5addd9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:31:22.624842 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:22.624804 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21d8d9-79e8-47d5-b135-556be5addd9e-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:31:23.383241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:23.383148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" event={"ID":"2c21d8d9-79e8-47d5-b135-556be5addd9e","Type":"ContainerDied","Data":"254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a"} Jul 15 08:31:23.383241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:23.383198 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254948a11150dbab18a66b1e0fe882a89d637f38880b081cf8dd6d40a7f72c0a" Jul 15 08:31:23.383241 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:23.383164 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93b7ed10-279x5" Jul 15 08:31:30.183115 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.183074 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:31:30.183642 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.183303 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" podUID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" containerName="maas-api" containerID="cri-o://0febbcf74e46c8905afbd4582ee8a9cbba48a62e52cd0479ffc32949c0badfc6" gracePeriod=30 Jul 15 08:31:30.439230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.439120 2565 generic.go:356] "Generic (PLEG): container finished" podID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" containerID="0febbcf74e46c8905afbd4582ee8a9cbba48a62e52cd0479ffc32949c0badfc6" exitCode=0 Jul 15 08:31:30.439230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.439193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" event={"ID":"e964cdca-f5ec-4fd0-94b8-ad622e44d93f","Type":"ContainerDied","Data":"0febbcf74e46c8905afbd4582ee8a9cbba48a62e52cd0479ffc32949c0badfc6"} Jul 15 08:31:30.439440 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.439320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" event={"ID":"e964cdca-f5ec-4fd0-94b8-ad622e44d93f","Type":"ContainerDied","Data":"359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de"} Jul 15 08:31:30.439440 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.439358 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="359c532a7438efdcade9b4891d128a2dedf34f64a85be183c241b64de7b1d6de" Jul 15 08:31:30.441983 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.441967 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:31:30.507498 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.507464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7w5x\" (UniqueName: \"kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x\") pod \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " Jul 15 08:31:30.507498 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.507500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls\") pod \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " Jul 15 08:31:30.507701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.507548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca\") pod \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\" (UID: \"e964cdca-f5ec-4fd0-94b8-ad622e44d93f\") " Jul 15 08:31:30.507996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.507964 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e964cdca-f5ec-4fd0-94b8-ad622e44d93f" (UID: "e964cdca-f5ec-4fd0-94b8-ad622e44d93f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:31:30.509585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.509561 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x" (OuterVolumeSpecName: "kube-api-access-n7w5x") pod "e964cdca-f5ec-4fd0-94b8-ad622e44d93f" (UID: "e964cdca-f5ec-4fd0-94b8-ad622e44d93f"). InnerVolumeSpecName "kube-api-access-n7w5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:31:30.509585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.509570 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e964cdca-f5ec-4fd0-94b8-ad622e44d93f" (UID: "e964cdca-f5ec-4fd0-94b8-ad622e44d93f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:31:30.609000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.608962 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7w5x\" (UniqueName: \"kubernetes.io/projected/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-kube-api-access-n7w5x\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:31:30.609000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.608995 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:31:30.609000 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:30.609004 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e964cdca-f5ec-4fd0-94b8-ad622e44d93f-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:31:31.444847 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:31.444758 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm" Jul 15 08:31:31.461165 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:31.461135 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:31:31.467006 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:31.466980 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7ed10-86f7978877-t9ccm"] Jul 15 08:31:33.117449 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:31:33.117389 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" path="/var/lib/kubelet/pods/e964cdca-f5ec-4fd0-94b8-ad622e44d93f/volumes" Jul 15 08:32:23.306999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:32:23.306965 2565 scope.go:117] "RemoveContainer" containerID="34a32f260b89930a0114115bdda73155da9e05a7fe29b69eb7f9436f8bf26786" Jul 15 08:33:29.966675 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.966638 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:29.967253 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967241 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:33:29.967332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967260 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:33:29.967332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967303 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c21d8d9-79e8-47d5-b135-556be5addd9e" containerName="revoke-keys" Jul 15 08:33:29.967332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967313 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c21d8d9-79e8-47d5-b135-556be5addd9e" containerName="revoke-keys" Jul 15 08:33:29.967332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967329 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" containerName="maas-api" Jul 15 08:33:29.967570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967338 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" containerName="maas-api" Jul 15 08:33:29.967570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967463 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e964cdca-f5ec-4fd0-94b8-ad622e44d93f" containerName="maas-api" Jul 15 08:33:29.967570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967480 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9d3a45-b59e-4c37-9b5e-04a17c185c51" containerName="revoke-keys" Jul 15 08:33:29.967570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.967489 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c21d8d9-79e8-47d5-b135-556be5addd9e" containerName="revoke-keys" Jul 15 08:33:29.970797 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.970776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:29.975127 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.975111 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-13831f8c-serving-cert\"" Jul 15 08:33:29.986946 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:29.986923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:30.092492 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.092460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq4q\" (UniqueName: \"kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.092681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.092523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.092681 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.092649 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.193277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.193247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq4q\" (UniqueName: \"kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.193476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.193310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.193476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.193366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.193613 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:33:30.193528 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-serving-cert: secret "maas-api-e2e-mt-13831f8c-serving-cert" not found Jul 15 08:33:30.193613 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:33:30.193595 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls podName:edcb39bd-63d9-48f2-9158-0a67afcfd54b nodeName:}" failed. No retries permitted until 2026-07-15 08:33:30.693575551 +0000 UTC m=+1988.113320346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls") pod "maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" (UID: "edcb39bd-63d9-48f2-9158-0a67afcfd54b") : secret "maas-api-e2e-mt-13831f8c-serving-cert" not found Jul 15 08:33:30.194077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.194049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.217720 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.217649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq4q\" (UniqueName: \"kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.699209 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.699175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.701652 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.701629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") pod \"maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:30.881132 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:30.881103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:31.027712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:31.027682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:31.028063 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:33:31.027937 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcb39bd_63d9_48f2_9158_0a67afcfd54b.slice/crio-69b7f05ac9a49f249b069791a0a29ac070237ddf74921736f6353467612db8cd WatchSource:0}: Error finding container 69b7f05ac9a49f249b069791a0a29ac070237ddf74921736f6353467612db8cd: Status 404 returned error can't find the container with id 69b7f05ac9a49f249b069791a0a29ac070237ddf74921736f6353467612db8cd Jul 15 08:33:31.029577 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:31.029560 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:33:31.095562 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:31.095528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" event={"ID":"edcb39bd-63d9-48f2-9158-0a67afcfd54b","Type":"ContainerStarted","Data":"69b7f05ac9a49f249b069791a0a29ac070237ddf74921736f6353467612db8cd"} Jul 15 08:33:32.102781 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:32.102745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" event={"ID":"edcb39bd-63d9-48f2-9158-0a67afcfd54b","Type":"ContainerStarted","Data":"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded"} Jul 15 08:33:32.103268 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:32.102814 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:32.130488 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:32.130443 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" podStartSLOduration=2.6951263020000003 podStartE2EDuration="3.130428997s" totalImagesPullingTime="435.302695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:33:32.129799646 +0000 UTC m=+1989.549544470" watchObservedRunningTime="2026-07-15 08:33:32.130428997 +0000 UTC m=+1989.550173800" Jul 15 08:33:38.113659 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:38.113630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:42.561717 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.561680 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj"] Jul 15 08:33:42.565637 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.565614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:42.569282 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.569263 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:33:42.582876 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.582848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj"] Jul 15 08:33:42.605594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.605568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj\" (UID: \"59b20ef3-68e5-4c4d-94c0-cccb92b6db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:42.706744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.706708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj\" (UID: \"59b20ef3-68e5-4c4d-94c0-cccb92b6db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:42.707371 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.707344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj\" (UID: \"59b20ef3-68e5-4c4d-94c0-cccb92b6db12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:42.875651 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:42.875562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:43.022109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:43.022083 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj"] Jul 15 08:33:43.024451 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:33:43.024421 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b20ef3_68e5_4c4d_94c0_cccb92b6db12.slice/crio-42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff WatchSource:0}: Error finding container 42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff: Status 404 returned error can't find the container with id 42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff Jul 15 08:33:43.161223 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:43.161134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" event={"ID":"59b20ef3-68e5-4c4d-94c0-cccb92b6db12","Type":"ContainerStarted","Data":"42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff"} Jul 15 08:33:44.177515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:44.177483 2565 generic.go:356] "Generic (PLEG): container finished" podID="59b20ef3-68e5-4c4d-94c0-cccb92b6db12" containerID="36910271570b2911d5ca633fb057b13cd7441cefd31e76551197d3f8bee91f13" exitCode=0 Jul 15 08:33:44.177919 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:44.177525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" event={"ID":"59b20ef3-68e5-4c4d-94c0-cccb92b6db12","Type":"ContainerDied","Data":"36910271570b2911d5ca633fb057b13cd7441cefd31e76551197d3f8bee91f13"} Jul 15 08:33:45.320542 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:45.320517 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:45.330410 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:45.330381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca\") pod \"59b20ef3-68e5-4c4d-94c0-cccb92b6db12\" (UID: \"59b20ef3-68e5-4c4d-94c0-cccb92b6db12\") " Jul 15 08:33:45.330707 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:45.330685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59b20ef3-68e5-4c4d-94c0-cccb92b6db12" (UID: "59b20ef3-68e5-4c4d-94c0-cccb92b6db12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:33:45.431203 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:45.431169 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b20ef3-68e5-4c4d-94c0-cccb92b6db12-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:33:46.190286 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:46.190256 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" Jul 15 08:33:46.190502 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:46.190286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj" event={"ID":"59b20ef3-68e5-4c4d-94c0-cccb92b6db12","Type":"ContainerDied","Data":"42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff"} Jul 15 08:33:46.190502 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:46.190331 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b2b93593f6d111ee4d513f128a0e72e00b380890e2142a243e316a612b8eff" Jul 15 08:33:53.275498 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.275459 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:53.275973 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.275683 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" podUID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" containerName="maas-api" containerID="cri-o://61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded" gracePeriod=30 Jul 15 08:33:53.535261 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.535186 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:53.610370 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.610326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrq4q\" (UniqueName: \"kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q\") pod \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " Jul 15 08:33:53.610589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.610412 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca\") pod \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " Jul 15 08:33:53.610589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.610463 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") pod \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\" (UID: \"edcb39bd-63d9-48f2-9158-0a67afcfd54b\") " Jul 15 08:33:53.610812 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.610785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edcb39bd-63d9-48f2-9158-0a67afcfd54b" (UID: "edcb39bd-63d9-48f2-9158-0a67afcfd54b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:33:53.612516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.612483 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q" (OuterVolumeSpecName: "kube-api-access-rrq4q") pod "edcb39bd-63d9-48f2-9158-0a67afcfd54b" (UID: "edcb39bd-63d9-48f2-9158-0a67afcfd54b"). InnerVolumeSpecName "kube-api-access-rrq4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:33:53.612624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.612530 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edcb39bd-63d9-48f2-9158-0a67afcfd54b" (UID: "edcb39bd-63d9-48f2-9158-0a67afcfd54b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:33:53.711921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.711885 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcb39bd-63d9-48f2-9158-0a67afcfd54b-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:33:53.711921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.711915 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq4q\" (UniqueName: \"kubernetes.io/projected/edcb39bd-63d9-48f2-9158-0a67afcfd54b-kube-api-access-rrq4q\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:33:53.711921 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:53.711925 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcb39bd-63d9-48f2-9158-0a67afcfd54b-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:33:54.237777 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.237740 2565 generic.go:356] "Generic (PLEG): container finished" podID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" containerID="61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded" exitCode=0 Jul 15 08:33:54.237947 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.237787 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" Jul 15 08:33:54.237947 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.237823 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" event={"ID":"edcb39bd-63d9-48f2-9158-0a67afcfd54b","Type":"ContainerDied","Data":"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded"} Jul 15 08:33:54.237947 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.237881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk" event={"ID":"edcb39bd-63d9-48f2-9158-0a67afcfd54b","Type":"ContainerDied","Data":"69b7f05ac9a49f249b069791a0a29ac070237ddf74921736f6353467612db8cd"} Jul 15 08:33:54.237947 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.237912 2565 scope.go:117] "RemoveContainer" containerID="61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded" Jul 15 08:33:54.250396 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.250366 2565 scope.go:117] "RemoveContainer" containerID="61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded" Jul 15 08:33:54.250889 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:33:54.250866 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded\": container with ID starting with 61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded not found: ID does not exist" containerID="61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded" Jul 15 08:33:54.250994 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.250901 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded"} err="failed to get container status \"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded\": rpc error: code = NotFound desc = could not find container \"61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded\": container with ID starting with 61c1c50f41651018ba079a32668c2332cbed3bb78905ebbe158529baab688ded not found: ID does not exist" Jul 15 08:33:54.263164 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.263128 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:54.265543 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:54.265516 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-13831f8c-7d84497fbc-dttxk"] Jul 15 08:33:55.116702 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:33:55.116670 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" path="/var/lib/kubelet/pods/edcb39bd-63d9-48f2-9158-0a67afcfd54b/volumes" Jul 15 08:34:23.339545 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:23.339511 2565 scope.go:117] "RemoveContainer" containerID="0499dcce7b0685d9084709d05e66a315cdf9cf018b0b846c388660e2181bb304" Jul 15 08:34:34.232906 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.232871 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:34:34.233288 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233277 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b20ef3-68e5-4c4d-94c0-cccb92b6db12" containerName="revoke-keys" Jul 15 08:34:34.233330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233289 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b20ef3-68e5-4c4d-94c0-cccb92b6db12" containerName="revoke-keys" Jul 15 08:34:34.233330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233305 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" containerName="maas-api" Jul 15 08:34:34.233330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233311 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" containerName="maas-api" Jul 15 08:34:34.233463 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233383 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b20ef3-68e5-4c4d-94c0-cccb92b6db12" containerName="revoke-keys" Jul 15 08:34:34.233463 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.233392 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcb39bd-63d9-48f2-9158-0a67afcfd54b" containerName="maas-api" Jul 15 08:34:34.236595 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.236579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.240654 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.240634 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2082f65c-serving-cert\"" Jul 15 08:34:34.249058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.249032 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:34:34.300495 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.300452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.300495 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.300493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkzd\" (UniqueName: \"kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.300723 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.300518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.401756 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.401718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.401756 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.401755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkzd\" (UniqueName: \"kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.401996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.401785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.401996 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:34:34.401878 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-serving-cert: secret "maas-api-e2e-mt-2082f65c-serving-cert" not found Jul 15 08:34:34.401996 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:34:34.401966 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls podName:6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94 nodeName:}" failed. No retries permitted until 2026-07-15 08:34:34.901944943 +0000 UTC m=+2052.321689725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls") pod "maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" (UID: "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94") : secret "maas-api-e2e-mt-2082f65c-serving-cert" not found Jul 15 08:34:34.402377 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.402360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.412617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.412585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkzd\" (UniqueName: \"kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.907150 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.907112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:34.909675 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:34.909651 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") pod \"maas-api-e2e-mt-2082f65c-54f6696784-2gbvc\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:35.147805 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:35.147766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:35.321292 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:35.321266 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:34:35.322571 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:34:35.322547 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3ea4ee_3bd0_44a0_b2c0_889b0f0a3d94.slice/crio-08d506c88d6744445000a0b505039f2894a64831a46abe9c4d042e90ce868a1b WatchSource:0}: Error finding container 08d506c88d6744445000a0b505039f2894a64831a46abe9c4d042e90ce868a1b: Status 404 returned error can't find the container with id 08d506c88d6744445000a0b505039f2894a64831a46abe9c4d042e90ce868a1b Jul 15 08:34:35.471624 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:35.471521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" event={"ID":"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94","Type":"ContainerStarted","Data":"08d506c88d6744445000a0b505039f2894a64831a46abe9c4d042e90ce868a1b"} Jul 15 08:34:36.478221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:36.478183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" event={"ID":"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94","Type":"ContainerStarted","Data":"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88"} Jul 15 08:34:36.478589 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:36.478331 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:34:36.498324 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:36.498273 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" podStartSLOduration=2.120734945 podStartE2EDuration="2.49825905s" totalImagesPullingTime="377.524105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:34:36.494936085 +0000 UTC m=+2053.914680889" watchObservedRunningTime="2026-07-15 08:34:36.49825905 +0000 UTC m=+2053.918003852" Jul 15 08:34:42.490629 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:34:42.490591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:35:17.070944 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.070852 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj"] Jul 15 08:35:17.075093 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.075066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:17.077669 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.077650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:35:17.083062 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.083034 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj"] Jul 15 08:35:17.220592 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.220551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj\" (UID: \"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:17.322638 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.322540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj\" (UID: \"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:17.323218 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.323199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj\" (UID: \"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:17.386346 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.386314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:17.537911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.537729 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj"] Jul 15 08:35:17.540697 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:35:17.540669 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d3a6c1_1ab3_434a_9ebf_4c2bfe0a1f54.slice/crio-a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712 WatchSource:0}: Error finding container a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712: Status 404 returned error can't find the container with id a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712 Jul 15 08:35:17.711828 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:17.711787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" event={"ID":"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54","Type":"ContainerStarted","Data":"a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712"} Jul 15 08:35:18.718559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:18.718463 2565 generic.go:356] "Generic (PLEG): container finished" podID="31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54" containerID="d4d3a89ef83c565de6927105fe90b927eec567fd28fde3b36938d8b6846fe8f0" exitCode=0 Jul 15 08:35:18.718916 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:18.718556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" event={"ID":"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54","Type":"ContainerDied","Data":"d4d3a89ef83c565de6927105fe90b927eec567fd28fde3b36938d8b6846fe8f0"} Jul 15 08:35:19.860995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:19.860969 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:19.948849 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:19.948808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca\") pod \"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54\" (UID: \"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54\") " Jul 15 08:35:19.949255 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:19.949226 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54" (UID: "31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:35:20.049829 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:20.049790 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:35:20.730874 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:20.730844 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" Jul 15 08:35:20.730874 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:20.730869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj" event={"ID":"31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54","Type":"ContainerDied","Data":"a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712"} Jul 15 08:35:20.731087 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:20.730916 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2dcb2df0b4dddce560f56a284652d262a5bbcc6deb3386c16579a4aee18b712" Jul 15 08:35:23.249211 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:23.249186 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:35:23.256181 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:23.256157 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:35:23.256740 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:23.256723 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:35:23.263648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:23.263630 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:35:23.366413 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:23.366368 2565 scope.go:117] "RemoveContainer" containerID="a30574635e07584951d7c0b1f31f64902e5afba6a800e98aca78de5576d177e1" Jul 15 08:35:27.794228 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:27.794188 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:35:27.794726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:27.794520 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" podUID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" containerName="maas-api" containerID="cri-o://80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88" gracePeriod=30 Jul 15 08:35:28.047776 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.047704 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:35:28.131292 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.131250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") pod \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " Jul 15 08:35:28.131506 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.131310 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca\") pod \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " Jul 15 08:35:28.131506 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.131349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnkzd\" (UniqueName: \"kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd\") pod \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\" (UID: \"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94\") " Jul 15 08:35:28.131701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.131678 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" (UID: "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:35:28.133493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.133466 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" (UID: "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:35:28.133493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.133479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd" (OuterVolumeSpecName: "kube-api-access-tnkzd") pod "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" (UID: "6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94"). InnerVolumeSpecName "kube-api-access-tnkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:35:28.232837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.232791 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:35:28.232837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.232829 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:35:28.232837 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.232838 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnkzd\" (UniqueName: \"kubernetes.io/projected/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94-kube-api-access-tnkzd\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:35:28.777960 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.777920 2565 generic.go:356] "Generic (PLEG): container finished" podID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" containerID="80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88" exitCode=0 Jul 15 08:35:28.778176 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.777986 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" Jul 15 08:35:28.778176 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.778003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" event={"ID":"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94","Type":"ContainerDied","Data":"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88"} Jul 15 08:35:28.778176 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.778084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc" event={"ID":"6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94","Type":"ContainerDied","Data":"08d506c88d6744445000a0b505039f2894a64831a46abe9c4d042e90ce868a1b"} Jul 15 08:35:28.778176 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.778108 2565 scope.go:117] "RemoveContainer" containerID="80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88" Jul 15 08:35:28.787116 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.787099 2565 scope.go:117] "RemoveContainer" containerID="80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88" Jul 15 08:35:28.787375 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:35:28.787357 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88\": container with ID starting with 80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88 not found: ID does not exist" containerID="80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88" Jul 15 08:35:28.787448 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.787385 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88"} err="failed to get container status \"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88\": rpc error: code = NotFound desc = could not find container \"80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88\": container with ID starting with 80eb3b3736118df9bd6b612b74c4fb681b37e893fc7d672d28f30a75688a7f88 not found: ID does not exist" Jul 15 08:35:28.800985 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.800954 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:35:28.804988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:28.804963 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2082f65c-54f6696784-2gbvc"] Jul 15 08:35:29.117792 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:35:29.117714 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" path="/var/lib/kubelet/pods/6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94/volumes" Jul 15 08:36:23.387701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:36:23.387672 2565 scope.go:117] "RemoveContainer" containerID="0febbcf74e46c8905afbd4582ee8a9cbba48a62e52cd0479ffc32949c0badfc6" Jul 15 08:37:23.412316 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:23.412283 2565 scope.go:117] "RemoveContainer" containerID="c2aaa3f3d2d276d606bda2ab4aef78295662fca1851c3a201a55f6f0573ec353" Jul 15 08:37:35.973248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973214 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:37:35.973651 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973637 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54" containerName="revoke-keys" Jul 15 08:37:35.973700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973652 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54" containerName="revoke-keys" Jul 15 08:37:35.973700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973663 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" containerName="maas-api" Jul 15 08:37:35.973700 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973669 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" containerName="maas-api" Jul 15 08:37:35.973792 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973740 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3ea4ee-3bd0-44a0-b2c0-889b0f0a3d94" containerName="maas-api" Jul 15 08:37:35.973792 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.973753 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54" containerName="revoke-keys" Jul 15 08:37:35.977043 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.977021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:35.980500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.980444 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f6b673-serving-cert\"" Jul 15 08:37:35.988305 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:35.988283 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:37:36.143795 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.143743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnhm\" (UniqueName: \"kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.143795 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.143794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.144100 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.143820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.244864 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.244822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnhm\" (UniqueName: \"kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.245077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.244882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.245077 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.244907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.245515 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.245494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.247504 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.247482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.254731 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.254704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnhm\" (UniqueName: \"kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm\") pod \"maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.288840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.288803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:36.432035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.432003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:37:36.433776 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:37:36.433748 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8ea547_1854_47ac_9801_eb9686b4ab39.slice/crio-e4022af8a804ead3e7e93b783260b437f8ec835a40c2f123bd9b8475de429982 WatchSource:0}: Error finding container e4022af8a804ead3e7e93b783260b437f8ec835a40c2f123bd9b8475de429982: Status 404 returned error can't find the container with id e4022af8a804ead3e7e93b783260b437f8ec835a40c2f123bd9b8475de429982 Jul 15 08:37:36.474972 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:36.474940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" event={"ID":"7b8ea547-1854-47ac-9801-eb9686b4ab39","Type":"ContainerStarted","Data":"e4022af8a804ead3e7e93b783260b437f8ec835a40c2f123bd9b8475de429982"} Jul 15 08:37:37.482062 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:37.482028 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" event={"ID":"7b8ea547-1854-47ac-9801-eb9686b4ab39","Type":"ContainerStarted","Data":"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6"} Jul 15 08:37:37.482489 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:37.482104 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:37.500994 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:37.500939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" podStartSLOduration=1.98739742 podStartE2EDuration="2.500920876s" totalImagesPullingTime="513.523456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:37:37.499047876 +0000 UTC m=+2234.918792678" watchObservedRunningTime="2026-07-15 08:37:37.500920876 +0000 UTC m=+2234.920665676" Jul 15 08:37:43.493244 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:43.493213 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:37:44.023686 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.023654 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:37:44.027516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.027493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.031330 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.031309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4c59e6-serving-cert\"" Jul 15 08:37:44.041057 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.041033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:37:44.121888 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.121846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6v2\" (UniqueName: \"kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.121888 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.121891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.122133 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.121932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.223094 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.223057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6v2\" (UniqueName: \"kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.223094 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.223101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.223312 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.223123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.223312 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:37:44.223214 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-serving-cert: secret "maas-api-e2e-inf-b-4c59e6-serving-cert" not found Jul 15 08:37:44.223312 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:37:44.223289 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls podName:7c75a079-b86c-459f-a1df-f8d38fb4cf39 nodeName:}" failed. No retries permitted until 2026-07-15 08:37:44.723269834 +0000 UTC m=+2242.143014615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls") pod "maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" (UID: "7c75a079-b86c-459f-a1df-f8d38fb4cf39") : secret "maas-api-e2e-inf-b-4c59e6-serving-cert" not found Jul 15 08:37:44.223800 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.223783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.246110 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.246076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6v2\" (UniqueName: \"kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.727846 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.727813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.730326 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.730292 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:44.938386 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:44.938348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:45.093192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:45.093169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:37:45.095127 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:37:45.095097 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c75a079_b86c_459f_a1df_f8d38fb4cf39.slice/crio-886c67e294899cd876e7dc04747a8f086acf0b196a7f19c1f476d1c5ce097e61 WatchSource:0}: Error finding container 886c67e294899cd876e7dc04747a8f086acf0b196a7f19c1f476d1c5ce097e61: Status 404 returned error can't find the container with id 886c67e294899cd876e7dc04747a8f086acf0b196a7f19c1f476d1c5ce097e61 Jul 15 08:37:45.526380 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:45.526340 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" event={"ID":"7c75a079-b86c-459f-a1df-f8d38fb4cf39","Type":"ContainerStarted","Data":"886c67e294899cd876e7dc04747a8f086acf0b196a7f19c1f476d1c5ce097e61"} Jul 15 08:37:46.533204 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:46.533164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" event={"ID":"7c75a079-b86c-459f-a1df-f8d38fb4cf39","Type":"ContainerStarted","Data":"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81"} Jul 15 08:37:46.533633 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:46.533267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:52.544382 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:52.544351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:37:52.566847 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:37:52.566785 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" podStartSLOduration=8.87625392 podStartE2EDuration="9.566766496s" totalImagesPullingTime="690.512576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:37:46.571999075 +0000 UTC m=+2243.991743878" watchObservedRunningTime="2026-07-15 08:37:52.566766496 +0000 UTC m=+2249.986511298" Jul 15 08:38:53.702621 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.702584 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt"] Jul 15 08:38:53.706373 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.706348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:53.709864 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.709840 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:38:53.727733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.727693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt"] Jul 15 08:38:53.813292 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.813255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt\" (UID: \"714071ef-cab4-4d86-ac4a-280a3c74ee4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:53.914435 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.914360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt\" (UID: \"714071ef-cab4-4d86-ac4a-280a3c74ee4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:53.915025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:53.915002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt\" (UID: \"714071ef-cab4-4d86-ac4a-280a3c74ee4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:54.017668 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:54.017633 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:54.183199 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:54.183172 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt"] Jul 15 08:38:54.185726 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:38:54.185697 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714071ef_cab4_4d86_ac4a_280a3c74ee4d.slice/crio-47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807 WatchSource:0}: Error finding container 47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807: Status 404 returned error can't find the container with id 47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807 Jul 15 08:38:54.897615 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:54.897586 2565 generic.go:356] "Generic (PLEG): container finished" podID="714071ef-cab4-4d86-ac4a-280a3c74ee4d" containerID="f1c17807b1c4f9b55b11816c6228043c7996ce470a81806a971b0b22c3f1134c" exitCode=0 Jul 15 08:38:54.897986 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:54.897660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" event={"ID":"714071ef-cab4-4d86-ac4a-280a3c74ee4d","Type":"ContainerDied","Data":"f1c17807b1c4f9b55b11816c6228043c7996ce470a81806a971b0b22c3f1134c"} Jul 15 08:38:54.897986 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:54.897711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" event={"ID":"714071ef-cab4-4d86-ac4a-280a3c74ee4d","Type":"ContainerStarted","Data":"47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807"} Jul 15 08:38:56.043739 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.043713 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:56.242736 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.242703 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca\") pod \"714071ef-cab4-4d86-ac4a-280a3c74ee4d\" (UID: \"714071ef-cab4-4d86-ac4a-280a3c74ee4d\") " Jul 15 08:38:56.243075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.243049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "714071ef-cab4-4d86-ac4a-280a3c74ee4d" (UID: "714071ef-cab4-4d86-ac4a-280a3c74ee4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:38:56.343493 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.343454 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714071ef-cab4-4d86-ac4a-280a3c74ee4d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:38:56.910818 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.910788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" Jul 15 08:38:56.910818 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.910796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt" event={"ID":"714071ef-cab4-4d86-ac4a-280a3c74ee4d","Type":"ContainerDied","Data":"47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807"} Jul 15 08:38:56.911086 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:38:56.910841 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a462880a91f85ec6d09bb6b6c75be9b42e1eb6f00f3057e3ff82a26b04f807" Jul 15 08:39:04.183035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.182996 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:39:04.183528 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.183229 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" podUID="7b8ea547-1854-47ac-9801-eb9686b4ab39" containerName="maas-api" containerID="cri-o://eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6" gracePeriod=30 Jul 15 08:39:04.449283 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.449213 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:39:04.512685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.512647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls\") pod \"7b8ea547-1854-47ac-9801-eb9686b4ab39\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " Jul 15 08:39:04.512894 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.512703 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mnhm\" (UniqueName: \"kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm\") pod \"7b8ea547-1854-47ac-9801-eb9686b4ab39\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " Jul 15 08:39:04.512894 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.512822 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca\") pod \"7b8ea547-1854-47ac-9801-eb9686b4ab39\" (UID: \"7b8ea547-1854-47ac-9801-eb9686b4ab39\") " Jul 15 08:39:04.513214 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.513181 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b8ea547-1854-47ac-9801-eb9686b4ab39" (UID: "7b8ea547-1854-47ac-9801-eb9686b4ab39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:39:04.514905 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.514875 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b8ea547-1854-47ac-9801-eb9686b4ab39" (UID: "7b8ea547-1854-47ac-9801-eb9686b4ab39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:39:04.515012 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.514908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm" (OuterVolumeSpecName: "kube-api-access-8mnhm") pod "7b8ea547-1854-47ac-9801-eb9686b4ab39" (UID: "7b8ea547-1854-47ac-9801-eb9686b4ab39"). InnerVolumeSpecName "kube-api-access-8mnhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:39:04.614011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.613967 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8ea547-1854-47ac-9801-eb9686b4ab39-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:04.614011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.613996 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mnhm\" (UniqueName: \"kubernetes.io/projected/7b8ea547-1854-47ac-9801-eb9686b4ab39-kube-api-access-8mnhm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:04.614011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.614015 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8ea547-1854-47ac-9801-eb9686b4ab39-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:04.956786 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.956753 2565 generic.go:356] "Generic (PLEG): container finished" podID="7b8ea547-1854-47ac-9801-eb9686b4ab39" containerID="eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6" exitCode=0 Jul 15 08:39:04.956975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.956800 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" Jul 15 08:39:04.956975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.956837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" event={"ID":"7b8ea547-1854-47ac-9801-eb9686b4ab39","Type":"ContainerDied","Data":"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6"} Jul 15 08:39:04.956975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.956892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24" event={"ID":"7b8ea547-1854-47ac-9801-eb9686b4ab39","Type":"ContainerDied","Data":"e4022af8a804ead3e7e93b783260b437f8ec835a40c2f123bd9b8475de429982"} Jul 15 08:39:04.956975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.956914 2565 scope.go:117] "RemoveContainer" containerID="eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6" Jul 15 08:39:04.966995 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.966971 2565 scope.go:117] "RemoveContainer" containerID="eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6" Jul 15 08:39:04.967303 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:39:04.967283 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6\": container with ID starting with eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6 not found: ID does not exist" containerID="eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6" Jul 15 08:39:04.967354 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.967317 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6"} err="failed to get container status \"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6\": rpc error: code = NotFound desc = could not find container \"eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6\": container with ID starting with eb0de17a0f68b351ccc1d0f2b0b1a1987030f68e878a1f23a74fd049b6b336c6 not found: ID does not exist" Jul 15 08:39:04.980335 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.980296 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:39:04.984221 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:04.984190 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6b673-58bcf4c458-n2c24"] Jul 15 08:39:05.117415 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:05.117373 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b8ea547-1854-47ac-9801-eb9686b4ab39" path="/var/lib/kubelet/pods/7b8ea547-1854-47ac-9801-eb9686b4ab39/volumes" Jul 15 08:39:19.925060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925026 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs"] Jul 15 08:39:19.925512 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925503 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b8ea547-1854-47ac-9801-eb9686b4ab39" containerName="maas-api" Jul 15 08:39:19.925559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925516 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b8ea547-1854-47ac-9801-eb9686b4ab39" containerName="maas-api" Jul 15 08:39:19.925559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925538 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714071ef-cab4-4d86-ac4a-280a3c74ee4d" containerName="revoke-keys" Jul 15 08:39:19.925559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925543 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="714071ef-cab4-4d86-ac4a-280a3c74ee4d" containerName="revoke-keys" Jul 15 08:39:19.925655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925614 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="714071ef-cab4-4d86-ac4a-280a3c74ee4d" containerName="revoke-keys" Jul 15 08:39:19.925655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.925627 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b8ea547-1854-47ac-9801-eb9686b4ab39" containerName="maas-api" Jul 15 08:39:19.928910 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.928894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:19.931555 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.931532 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:39:19.937500 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:19.937478 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs"] Jul 15 08:39:20.066767 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:20.066719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs\" (UID: \"9b9dd329-ccc8-48c9-9868-16ac89623c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:20.167456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:20.167393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs\" (UID: \"9b9dd329-ccc8-48c9-9868-16ac89623c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:20.168109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:20.168088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs\" (UID: \"9b9dd329-ccc8-48c9-9868-16ac89623c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:20.242192 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:20.242157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:20.380475 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:20.380449 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs"] Jul 15 08:39:20.382300 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:39:20.382272 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9dd329_ccc8_48c9_9868_16ac89623c72.slice/crio-19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109 WatchSource:0}: Error finding container 19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109: Status 404 returned error can't find the container with id 19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109 Jul 15 08:39:21.064547 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:21.064506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" event={"ID":"9b9dd329-ccc8-48c9-9868-16ac89623c72","Type":"ContainerStarted","Data":"5c0284b3a68f47d327e40fc7ab448c56432c78fe1c12770f224b90d5e204f789"} Jul 15 08:39:21.064965 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:21.064589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" event={"ID":"9b9dd329-ccc8-48c9-9868-16ac89623c72","Type":"ContainerStarted","Data":"19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109"} Jul 15 08:39:21.083494 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:21.083444 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" podStartSLOduration=2.083426851 podStartE2EDuration="2.083426851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:39:21.081488169 +0000 UTC m=+2338.501232973" watchObservedRunningTime="2026-07-15 08:39:21.083426851 +0000 UTC m=+2338.503171654" Jul 15 08:39:22.071481 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:22.071437 2565 generic.go:356] "Generic (PLEG): container finished" podID="9b9dd329-ccc8-48c9-9868-16ac89623c72" containerID="5c0284b3a68f47d327e40fc7ab448c56432c78fe1c12770f224b90d5e204f789" exitCode=0 Jul 15 08:39:22.071888 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:22.071521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" event={"ID":"9b9dd329-ccc8-48c9-9868-16ac89623c72","Type":"ContainerDied","Data":"5c0284b3a68f47d327e40fc7ab448c56432c78fe1c12770f224b90d5e204f789"} Jul 15 08:39:23.216016 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:23.215985 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:23.298025 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:23.297972 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca\") pod \"9b9dd329-ccc8-48c9-9868-16ac89623c72\" (UID: \"9b9dd329-ccc8-48c9-9868-16ac89623c72\") " Jul 15 08:39:23.298358 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:23.298333 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b9dd329-ccc8-48c9-9868-16ac89623c72" (UID: "9b9dd329-ccc8-48c9-9868-16ac89623c72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:39:23.399163 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:23.399074 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9dd329-ccc8-48c9-9868-16ac89623c72-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:24.083234 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:24.083198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" event={"ID":"9b9dd329-ccc8-48c9-9868-16ac89623c72","Type":"ContainerDied","Data":"19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109"} Jul 15 08:39:24.083234 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:24.083228 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs" Jul 15 08:39:24.083482 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:24.083250 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19eab639d9cde494bd1f23543913c5d9b344da73fbc66f2b7fc62f44cb0b7109" Jul 15 08:39:30.430852 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.430814 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:39:30.431377 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.431076 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" podUID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" containerName="maas-api" containerID="cri-o://94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81" gracePeriod=30 Jul 15 08:39:30.687060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.686996 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:39:30.877518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.877468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") pod \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " Jul 15 08:39:30.877699 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.877537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca\") pod \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " Jul 15 08:39:30.877746 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.877695 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f6v2\" (UniqueName: \"kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2\") pod \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\" (UID: \"7c75a079-b86c-459f-a1df-f8d38fb4cf39\") " Jul 15 08:39:30.877879 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.877854 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c75a079-b86c-459f-a1df-f8d38fb4cf39" (UID: "7c75a079-b86c-459f-a1df-f8d38fb4cf39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:39:30.878049 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.878026 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c75a079-b86c-459f-a1df-f8d38fb4cf39-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:30.879761 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.879735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2" (OuterVolumeSpecName: "kube-api-access-2f6v2") pod "7c75a079-b86c-459f-a1df-f8d38fb4cf39" (UID: "7c75a079-b86c-459f-a1df-f8d38fb4cf39"). InnerVolumeSpecName "kube-api-access-2f6v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:39:30.879761 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.879755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c75a079-b86c-459f-a1df-f8d38fb4cf39" (UID: "7c75a079-b86c-459f-a1df-f8d38fb4cf39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:39:30.978954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.978862 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c75a079-b86c-459f-a1df-f8d38fb4cf39-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:30.978954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:30.978893 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f6v2\" (UniqueName: \"kubernetes.io/projected/7c75a079-b86c-459f-a1df-f8d38fb4cf39-kube-api-access-2f6v2\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:39:31.125911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.125870 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" containerID="94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81" exitCode=0 Jul 15 08:39:31.126060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.125935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" event={"ID":"7c75a079-b86c-459f-a1df-f8d38fb4cf39","Type":"ContainerDied","Data":"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81"} Jul 15 08:39:31.126060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.125953 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" Jul 15 08:39:31.126060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.126015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss" event={"ID":"7c75a079-b86c-459f-a1df-f8d38fb4cf39","Type":"ContainerDied","Data":"886c67e294899cd876e7dc04747a8f086acf0b196a7f19c1f476d1c5ce097e61"} Jul 15 08:39:31.126060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.126049 2565 scope.go:117] "RemoveContainer" containerID="94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81" Jul 15 08:39:31.135436 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.135389 2565 scope.go:117] "RemoveContainer" containerID="94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81" Jul 15 08:39:31.135744 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:39:31.135722 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81\": container with ID starting with 94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81 not found: ID does not exist" containerID="94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81" Jul 15 08:39:31.135836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.135752 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81"} err="failed to get container status \"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81\": rpc error: code = NotFound desc = could not find container \"94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81\": container with ID starting with 94746af125aa8f004b3a86559a8b28ceb5a33f99202a9cc3e6a477fb03b03d81 not found: ID does not exist" Jul 15 08:39:31.145651 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.145600 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:39:31.148247 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:31.148225 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c59e6-645bdcb9d7-kbgss"] Jul 15 08:39:33.116640 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:33.116606 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" path="/var/lib/kubelet/pods/7c75a079-b86c-459f-a1df-f8d38fb4cf39/volumes" Jul 15 08:39:49.856153 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856115 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:39:49.856808 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856791 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9dd329-ccc8-48c9-9868-16ac89623c72" containerName="revoke-keys" Jul 15 08:39:49.856860 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856811 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9dd329-ccc8-48c9-9868-16ac89623c72" containerName="revoke-keys" Jul 15 08:39:49.856860 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856829 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" containerName="maas-api" Jul 15 08:39:49.856860 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856838 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" containerName="maas-api" Jul 15 08:39:49.856977 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856965 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9dd329-ccc8-48c9-9868-16ac89623c72" containerName="revoke-keys" Jul 15 08:39:49.857019 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.856984 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c75a079-b86c-459f-a1df-f8d38fb4cf39" containerName="maas-api" Jul 15 08:39:49.862058 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.862031 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:49.864454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.864395 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ec03f1-serving-cert\"" Jul 15 08:39:49.870189 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.870162 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:39:49.975872 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.975834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24ph\" (UniqueName: \"kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:49.976129 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.975968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:49.976129 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:49.976014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.077377 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.077337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.077377 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.077382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.077637 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.077506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24ph\" (UniqueName: \"kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.077637 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:39:50.077525 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-serving-cert: secret "maas-api-e2e-api-a-ec03f1-serving-cert" not found Jul 15 08:39:50.077637 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:39:50.077603 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls podName:653e8ff9-13ca-46b0-84c1-72a9562709e2 nodeName:}" failed. No retries permitted until 2026-07-15 08:39:50.577586738 +0000 UTC m=+2367.997331523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls") pod "maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" (UID: "653e8ff9-13ca-46b0-84c1-72a9562709e2") : secret "maas-api-e2e-api-a-ec03f1-serving-cert" not found Jul 15 08:39:50.078038 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.078019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.092273 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.092238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24ph\" (UniqueName: \"kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.583976 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.583938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.586513 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.586486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") pod \"maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.774938 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.774902 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:50.925189 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.925157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:39:50.926033 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:39:50.925993 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653e8ff9_13ca_46b0_84c1_72a9562709e2.slice/crio-20ef1b8b2fba482234375280d68722fe65d4f92cbccb3e4f61192b8ee74cf5a7 WatchSource:0}: Error finding container 20ef1b8b2fba482234375280d68722fe65d4f92cbccb3e4f61192b8ee74cf5a7: Status 404 returned error can't find the container with id 20ef1b8b2fba482234375280d68722fe65d4f92cbccb3e4f61192b8ee74cf5a7 Jul 15 08:39:50.927454 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:50.927433 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 08:39:51.245253 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:51.245216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" event={"ID":"653e8ff9-13ca-46b0-84c1-72a9562709e2","Type":"ContainerStarted","Data":"20ef1b8b2fba482234375280d68722fe65d4f92cbccb3e4f61192b8ee74cf5a7"} Jul 15 08:39:52.253164 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:52.253124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" event={"ID":"653e8ff9-13ca-46b0-84c1-72a9562709e2","Type":"ContainerStarted","Data":"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45"} Jul 15 08:39:52.253606 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:52.253200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:52.272047 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:52.271994 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" podStartSLOduration=2.83500884 podStartE2EDuration="3.271976461s" totalImagesPullingTime="436.967621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:39:52.270664584 +0000 UTC m=+2369.690409388" watchObservedRunningTime="2026-07-15 08:39:52.271976461 +0000 UTC m=+2369.691721312" Jul 15 08:39:57.899748 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:57.899707 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:39:57.903905 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:57.903880 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:57.906898 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:57.906873 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7a0e41-serving-cert\"" Jul 15 08:39:57.916349 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:57.916318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:39:58.056810 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.056768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.056999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.056818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.056999 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.056962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv26m\" (UniqueName: \"kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.157566 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.157475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.157723 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.157582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv26m\" (UniqueName: \"kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.157723 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.157617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.158261 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.158237 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.160024 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.160006 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.166744 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.166722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv26m\" (UniqueName: \"kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m\") pod \"maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.215690 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.215631 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:58.264759 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.264731 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:39:58.400032 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:58.400004 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:39:58.401611 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:39:58.401584 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bb00b5_ab4b_4c57_aca0_3665ad78008c.slice/crio-6ae8938121f73d3561540e85e67a0eae55439ad304ce96b53227a2a7f3d9ff3b WatchSource:0}: Error finding container 6ae8938121f73d3561540e85e67a0eae55439ad304ce96b53227a2a7f3d9ff3b: Status 404 returned error can't find the container with id 6ae8938121f73d3561540e85e67a0eae55439ad304ce96b53227a2a7f3d9ff3b Jul 15 08:39:59.295739 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:59.295707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" event={"ID":"52bb00b5-ab4b-4c57-aca0-3665ad78008c","Type":"ContainerStarted","Data":"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48"} Jul 15 08:39:59.296145 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:59.295763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" event={"ID":"52bb00b5-ab4b-4c57-aca0-3665ad78008c","Type":"ContainerStarted","Data":"6ae8938121f73d3561540e85e67a0eae55439ad304ce96b53227a2a7f3d9ff3b"} Jul 15 08:39:59.296145 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:59.295790 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:39:59.320361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:39:59.320298 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" podStartSLOduration=1.965816048 podStartE2EDuration="2.320276403s" totalImagesPullingTime="354.460355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:39:59.319646116 +0000 UTC m=+2376.739390933" watchObservedRunningTime="2026-07-15 08:39:59.320276403 +0000 UTC m=+2376.740021207" Jul 15 08:40:05.306651 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:05.306620 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:40:10.778166 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.778128 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf"] Jul 15 08:40:10.782379 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.782354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:10.785015 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.784988 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:40:10.792231 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.792200 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf"] Jul 15 08:40:10.878081 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.878035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf\" (UID: \"837072e9-be6f-4867-823f-4388a766bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:10.979123 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.979085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf\" (UID: \"837072e9-be6f-4867-823f-4388a766bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:10.979933 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:10.979909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf\" (UID: \"837072e9-be6f-4867-823f-4388a766bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:11.096156 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:11.096050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:11.247930 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:11.247896 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf"] Jul 15 08:40:11.256111 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:40:11.256083 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837072e9_be6f_4867_823f_4388a766bab8.slice/crio-f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4 WatchSource:0}: Error finding container f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4: Status 404 returned error can't find the container with id f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4 Jul 15 08:40:11.364335 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:11.364246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" event={"ID":"837072e9-be6f-4867-823f-4388a766bab8","Type":"ContainerStarted","Data":"f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4"} Jul 15 08:40:12.372179 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:12.372090 2565 generic.go:356] "Generic (PLEG): container finished" podID="837072e9-be6f-4867-823f-4388a766bab8" containerID="d4dafdb56329558e01e75a56d3ac141025c3ea6b32021b568124f594de32f2a8" exitCode=0 Jul 15 08:40:12.372179 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:12.372143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" event={"ID":"837072e9-be6f-4867-823f-4388a766bab8","Type":"ContainerDied","Data":"d4dafdb56329558e01e75a56d3ac141025c3ea6b32021b568124f594de32f2a8"} Jul 15 08:40:13.514830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:13.514794 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:13.605100 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:13.605068 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca\") pod \"837072e9-be6f-4867-823f-4388a766bab8\" (UID: \"837072e9-be6f-4867-823f-4388a766bab8\") " Jul 15 08:40:13.605467 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:13.605446 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "837072e9-be6f-4867-823f-4388a766bab8" (UID: "837072e9-be6f-4867-823f-4388a766bab8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:40:13.706645 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:13.706568 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837072e9-be6f-4867-823f-4388a766bab8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:14.384137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:14.384107 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" Jul 15 08:40:14.384317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:14.384143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf" event={"ID":"837072e9-be6f-4867-823f-4388a766bab8","Type":"ContainerDied","Data":"f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4"} Jul 15 08:40:14.384317 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:14.384191 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f499cf737e9c0de42c927a3a8ea38072b6e5aefc701bbc7ce777ffa075a006c4" Jul 15 08:40:21.758083 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:21.758042 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:40:21.758923 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:21.758888 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" podUID="653e8ff9-13ca-46b0-84c1-72a9562709e2" containerName="maas-api" containerID="cri-o://eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45" gracePeriod=30 Jul 15 08:40:22.014795 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.014729 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:40:22.092822 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.092789 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") pod \"653e8ff9-13ca-46b0-84c1-72a9562709e2\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " Jul 15 08:40:22.092989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.092830 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca\") pod \"653e8ff9-13ca-46b0-84c1-72a9562709e2\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " Jul 15 08:40:22.092989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.092912 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p24ph\" (UniqueName: \"kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph\") pod \"653e8ff9-13ca-46b0-84c1-72a9562709e2\" (UID: \"653e8ff9-13ca-46b0-84c1-72a9562709e2\") " Jul 15 08:40:22.093230 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.093209 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "653e8ff9-13ca-46b0-84c1-72a9562709e2" (UID: "653e8ff9-13ca-46b0-84c1-72a9562709e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:40:22.095011 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.094986 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph" (OuterVolumeSpecName: "kube-api-access-p24ph") pod "653e8ff9-13ca-46b0-84c1-72a9562709e2" (UID: "653e8ff9-13ca-46b0-84c1-72a9562709e2"). InnerVolumeSpecName "kube-api-access-p24ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:40:22.095076 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.095055 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "653e8ff9-13ca-46b0-84c1-72a9562709e2" (UID: "653e8ff9-13ca-46b0-84c1-72a9562709e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:40:22.194741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.194703 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24ph\" (UniqueName: \"kubernetes.io/projected/653e8ff9-13ca-46b0-84c1-72a9562709e2-kube-api-access-p24ph\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:22.194741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.194735 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653e8ff9-13ca-46b0-84c1-72a9562709e2-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:22.194741 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.194744 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653e8ff9-13ca-46b0-84c1-72a9562709e2-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:22.431135 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.431045 2565 generic.go:356] "Generic (PLEG): container finished" podID="653e8ff9-13ca-46b0-84c1-72a9562709e2" containerID="eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45" exitCode=0 Jul 15 08:40:22.431135 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.431095 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" Jul 15 08:40:22.431323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.431130 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" event={"ID":"653e8ff9-13ca-46b0-84c1-72a9562709e2","Type":"ContainerDied","Data":"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45"} Jul 15 08:40:22.431323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.431184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j" event={"ID":"653e8ff9-13ca-46b0-84c1-72a9562709e2","Type":"ContainerDied","Data":"20ef1b8b2fba482234375280d68722fe65d4f92cbccb3e4f61192b8ee74cf5a7"} Jul 15 08:40:22.431323 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.431205 2565 scope.go:117] "RemoveContainer" containerID="eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45" Jul 15 08:40:22.441245 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.441228 2565 scope.go:117] "RemoveContainer" containerID="eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45" Jul 15 08:40:22.441554 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:40:22.441533 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45\": container with ID starting with eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45 not found: ID does not exist" containerID="eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45" Jul 15 08:40:22.441626 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.441564 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45"} err="failed to get container status \"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45\": rpc error: code = NotFound desc = could not find container \"eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45\": container with ID starting with eeed53a85006f6117fd8b1c08445662bee5e02b92811839de2fb37bba10efb45 not found: ID does not exist" Jul 15 08:40:22.454769 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.454740 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:40:22.456968 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:22.456949 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec03f1-74bc594fd5-5rm2j"] Jul 15 08:40:23.117116 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:23.117080 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="653e8ff9-13ca-46b0-84c1-72a9562709e2" path="/var/lib/kubelet/pods/653e8ff9-13ca-46b0-84c1-72a9562709e2/volumes" Jul 15 08:40:23.285671 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:23.285568 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:40:23.312339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:23.300053 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:40:23.312339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:23.301237 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:40:23.312339 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:23.308542 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:40:37.050836 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.050797 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285"] Jul 15 08:40:37.051301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837072e9-be6f-4867-823f-4388a766bab8" containerName="revoke-keys" Jul 15 08:40:37.051301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051226 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="837072e9-be6f-4867-823f-4388a766bab8" containerName="revoke-keys" Jul 15 08:40:37.051301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051246 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653e8ff9-13ca-46b0-84c1-72a9562709e2" containerName="maas-api" Jul 15 08:40:37.051301 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051253 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="653e8ff9-13ca-46b0-84c1-72a9562709e2" containerName="maas-api" Jul 15 08:40:37.051497 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051320 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="653e8ff9-13ca-46b0-84c1-72a9562709e2" containerName="maas-api" Jul 15 08:40:37.051497 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.051329 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="837072e9-be6f-4867-823f-4388a766bab8" containerName="revoke-keys" Jul 15 08:40:37.054651 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.054633 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:37.057307 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.057285 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:40:37.062648 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.062618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285"] Jul 15 08:40:37.141833 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.141799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a0e41-x5285\" (UID: \"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:37.243463 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.243397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a0e41-x5285\" (UID: \"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:37.244095 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.244070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7a0e41-x5285\" (UID: \"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:37.367099 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.367006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:37.507689 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.507658 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285"] Jul 15 08:40:37.509692 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:40:37.509664 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff0bb17_fd41_4277_9fb2_ea0d3bb3cf26.slice/crio-f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005 WatchSource:0}: Error finding container f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005: Status 404 returned error can't find the container with id f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005 Jul 15 08:40:37.519969 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:37.519943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" event={"ID":"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26","Type":"ContainerStarted","Data":"f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005"} Jul 15 08:40:38.527668 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:38.527634 2565 generic.go:356] "Generic (PLEG): container finished" podID="4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26" containerID="6da10a95d1e56cc93eec718059eec4df1b9f38334d6b80ea794a4ae74f3a9339" exitCode=0 Jul 15 08:40:38.528075 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:38.527717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" event={"ID":"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26","Type":"ContainerDied","Data":"6da10a95d1e56cc93eec718059eec4df1b9f38334d6b80ea794a4ae74f3a9339"} Jul 15 08:40:39.669883 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:39.669856 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:39.769390 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:39.769353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca\") pod \"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26\" (UID: \"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26\") " Jul 15 08:40:39.769726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:39.769704 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26" (UID: "4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:40:39.870996 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:39.870904 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:40.539602 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:40.539573 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" Jul 15 08:40:40.539602 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:40.539586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7a0e41-x5285" event={"ID":"4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26","Type":"ContainerDied","Data":"f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005"} Jul 15 08:40:40.539813 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:40.539633 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5949ec3b99e997ddcb6b4c91e8c3717bb7b7c3b75ea0bcf7f773b600ae80005" Jul 15 08:40:47.861677 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:47.861595 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:40:47.862117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:47.861792 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" podUID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" containerName="maas-api" containerID="cri-o://ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48" gracePeriod=30 Jul 15 08:40:48.119552 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.119469 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:40:48.149583 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.149552 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca\") pod \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " Jul 15 08:40:48.149791 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.149662 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls\") pod \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " Jul 15 08:40:48.149791 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.149694 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv26m\" (UniqueName: \"kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m\") pod \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\" (UID: \"52bb00b5-ab4b-4c57-aca0-3665ad78008c\") " Jul 15 08:40:48.150060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.150032 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52bb00b5-ab4b-4c57-aca0-3665ad78008c" (UID: "52bb00b5-ab4b-4c57-aca0-3665ad78008c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:40:48.151875 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.151847 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52bb00b5-ab4b-4c57-aca0-3665ad78008c" (UID: "52bb00b5-ab4b-4c57-aca0-3665ad78008c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:40:48.151975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.151918 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m" (OuterVolumeSpecName: "kube-api-access-pv26m") pod "52bb00b5-ab4b-4c57-aca0-3665ad78008c" (UID: "52bb00b5-ab4b-4c57-aca0-3665ad78008c"). InnerVolumeSpecName "kube-api-access-pv26m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:40:48.250518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.250466 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52bb00b5-ab4b-4c57-aca0-3665ad78008c-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:48.250518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.250507 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52bb00b5-ab4b-4c57-aca0-3665ad78008c-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:48.250518 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.250518 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv26m\" (UniqueName: \"kubernetes.io/projected/52bb00b5-ab4b-4c57-aca0-3665ad78008c-kube-api-access-pv26m\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:40:48.589098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.589059 2565 generic.go:356] "Generic (PLEG): container finished" podID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" containerID="ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48" exitCode=0 Jul 15 08:40:48.589098 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.589103 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" Jul 15 08:40:48.589361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.589132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" event={"ID":"52bb00b5-ab4b-4c57-aca0-3665ad78008c","Type":"ContainerDied","Data":"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48"} Jul 15 08:40:48.589361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.589182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh" event={"ID":"52bb00b5-ab4b-4c57-aca0-3665ad78008c","Type":"ContainerDied","Data":"6ae8938121f73d3561540e85e67a0eae55439ad304ce96b53227a2a7f3d9ff3b"} Jul 15 08:40:48.589361 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.589204 2565 scope.go:117] "RemoveContainer" containerID="ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48" Jul 15 08:40:48.599484 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.599463 2565 scope.go:117] "RemoveContainer" containerID="ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48" Jul 15 08:40:48.599773 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:40:48.599757 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48\": container with ID starting with ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48 not found: ID does not exist" containerID="ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48" Jul 15 08:40:48.599822 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.599783 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48"} err="failed to get container status \"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48\": rpc error: code = NotFound desc = could not find container \"ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48\": container with ID starting with ccba194303bcc383e827b8f8e4bcef1bb9f69bbf44e64c412dd430ec999d5e48 not found: ID does not exist" Jul 15 08:40:48.613307 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.613276 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:40:48.616713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:48.616688 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a0e41-57b7466c74-r2mlh"] Jul 15 08:40:49.116885 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:40:49.116850 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" path="/var/lib/kubelet/pods/52bb00b5-ab4b-4c57-aca0-3665ad78008c/volumes" Jul 15 08:41:03.971682 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.971649 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:41:03.972094 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972082 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26" containerName="revoke-keys" Jul 15 08:41:03.972094 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972095 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26" containerName="revoke-keys" Jul 15 08:41:03.972171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972108 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" containerName="maas-api" Jul 15 08:41:03.972171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972113 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" containerName="maas-api" Jul 15 08:41:03.972237 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972179 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26" containerName="revoke-keys" Jul 15 08:41:03.972237 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.972187 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bb00b5-ab4b-4c57-aca0-3665ad78008c" containerName="maas-api" Jul 15 08:41:03.975375 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.975359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:03.978173 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.978151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8e59d0-openshift-default-dockercfg-lf9kp\"" Jul 15 08:41:03.988761 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:03.988737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:41:04.104279 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104511 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xpxs\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104511 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104511 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104638 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104638 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.104712 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.104703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206072 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206072 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xpxs\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206314 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206617 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206832 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.206832 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.206734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.207036 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.207011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.208757 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.208736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.208934 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.208916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.215348 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.215317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.215476 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.215467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xpxs\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs\") pod \"e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.286213 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.286171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:04.430773 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.430744 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:41:04.432205 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:41:04.432174 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3420a8_13cc_4c2d_a930_243c3598c2d0.slice/crio-1af3b7fa34f8065928a9058d892cf3c4742122882d761563273be9932ed2d279 WatchSource:0}: Error finding container 1af3b7fa34f8065928a9058d892cf3c4742122882d761563273be9932ed2d279: Status 404 returned error can't find the container with id 1af3b7fa34f8065928a9058d892cf3c4742122882d761563273be9932ed2d279 Jul 15 08:41:04.678067 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:04.677967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerStarted","Data":"1af3b7fa34f8065928a9058d892cf3c4742122882d761563273be9932ed2d279"} Jul 15 08:41:06.878142 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:06.878102 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:06.878441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:06.878180 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:06.878441 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:06.878208 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:07.697938 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:07.697901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerStarted","Data":"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa"} Jul 15 08:41:07.723186 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:07.723124 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podStartSLOduration=2.27973928 podStartE2EDuration="4.72310627s" totalImagesPullingTime="2.44336699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:41:07.720172521 +0000 UTC m=+2445.139917377" watchObservedRunningTime="2026-07-15 08:41:07.72310627 +0000 UTC m=+2445.142851073" Jul 15 08:41:08.286420 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:08.286364 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:08.291637 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:08.291608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:08.704186 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:08.704103 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:08.705069 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:08.705050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:18.146456 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.146420 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:41:18.151664 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.151641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.154293 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.154265 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1db148-serving-cert\"" Jul 15 08:41:18.159863 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.159831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:41:18.249690 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.249653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7swp\" (UniqueName: \"kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.249880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.249737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.249880 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.249775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.350962 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.350929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7swp\" (UniqueName: \"kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.351134 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.350997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.351134 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.351032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.351134 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:41:18.351131 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-serving-cert: secret "maas-api-e2e-shared-b-1db148-serving-cert" not found Jul 15 08:41:18.351266 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:41:18.351182 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls podName:67e4ef85-923e-4aa9-84e8-98c6b470e9e9 nodeName:}" failed. No retries permitted until 2026-07-15 08:41:18.851167498 +0000 UTC m=+2456.270912278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls") pod "maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" (UID: "67e4ef85-923e-4aa9-84e8-98c6b470e9e9") : secret "maas-api-e2e-shared-b-1db148-serving-cert" not found Jul 15 08:41:18.351752 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.351727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.362345 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.362315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7swp\" (UniqueName: \"kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.855927 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.855892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:18.858343 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:18.858315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:19.065300 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.065258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:19.235448 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.235420 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:41:19.238132 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:41:19.238101 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e4ef85_923e_4aa9_84e8_98c6b470e9e9.slice/crio-88b4f6f07aa7ca89834f283714d2bb372eaca9ca0bb10d5b7960b8a55172808e WatchSource:0}: Error finding container 88b4f6f07aa7ca89834f283714d2bb372eaca9ca0bb10d5b7960b8a55172808e: Status 404 returned error can't find the container with id 88b4f6f07aa7ca89834f283714d2bb372eaca9ca0bb10d5b7960b8a55172808e Jul 15 08:41:19.769696 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.769660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" event={"ID":"67e4ef85-923e-4aa9-84e8-98c6b470e9e9","Type":"ContainerStarted","Data":"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2"} Jul 15 08:41:19.769848 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.769711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" event={"ID":"67e4ef85-923e-4aa9-84e8-98c6b470e9e9","Type":"ContainerStarted","Data":"88b4f6f07aa7ca89834f283714d2bb372eaca9ca0bb10d5b7960b8a55172808e"} Jul 15 08:41:19.769848 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.769755 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:19.790439 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:19.790357 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" podStartSLOduration=1.453752934 podStartE2EDuration="1.790336305s" totalImagesPullingTime="336.583371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:41:19.787014245 +0000 UTC m=+2457.206759047" watchObservedRunningTime="2026-07-15 08:41:19.790336305 +0000 UTC m=+2457.210081109" Jul 15 08:41:21.781556 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.781524 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerID="0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa" exitCode=0 Jul 15 08:41:21.781989 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.781599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerDied","Data":"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa"} Jul 15 08:41:21.782045 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.782035 2565 scope.go:117] "RemoveContainer" containerID="0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa" Jul 15 08:41:21.784475 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.784442 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:21.784560 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.784517 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:21.784560 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:21.784549 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 08:41:22.789007 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:22.788967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerStarted","Data":"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce"} Jul 15 08:41:23.286594 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:23.286552 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:23.287981 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:23.287946 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 08:41:23.288121 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:23.288005 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 08:41:23.704682 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:23.704577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:24.288613 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:24.288561 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 08:41:24.289154 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:24.288648 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 08:41:25.287026 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:25.286980 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 08:41:25.287354 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:25.287319 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 08:41:25.787980 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:25.787724 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:41:26.287504 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:26.287460 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 08:41:26.287815 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:26.287786 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 08:41:27.287130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:27.287090 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 08:41:27.287775 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:27.287736 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 08:41:28.291029 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:28.290993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:41:28.291963 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:41:28.291944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:45:00.145621 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.145587 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc"] Jul 15 08:45:00.149718 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.149699 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j"] Jul 15 08:45:00.149833 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.149811 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" Jul 15 08:45:00.152942 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.152920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5f62w\"" Jul 15 08:45:00.153559 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.153540 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x"] Jul 15 08:45:00.153664 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.153644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" Jul 15 08:45:00.159201 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.159178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" Jul 15 08:45:00.168072 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.168047 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc"] Jul 15 08:45:00.169495 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.169474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" Jul 15 08:45:00.172537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.172518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x"] Jul 15 08:45:00.175477 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.175452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j"] Jul 15 08:45:00.184674 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.184650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" Jul 15 08:45:00.222671 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.222643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" Jul 15 08:45:00.340516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.340486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc"] Jul 15 08:45:00.341393 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:45:00.341353 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d696b34_a366_4051_ac36_d3b64807e467.slice/crio-76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27 WatchSource:0}: Error finding container 76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27: Status 404 returned error can't find the container with id 76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27 Jul 15 08:45:00.374797 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.374762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j"] Jul 15 08:45:00.379782 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:45:00.379751 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dce0f91_5618_4196_b403_cc15f22eb06f.slice/crio-8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4 WatchSource:0}: Error finding container 8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4: Status 404 returned error can't find the container with id 8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4 Jul 15 08:45:00.400830 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:00.400805 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x"] Jul 15 08:45:00.403061 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:45:00.403039 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6eadcda_60f3_4aec_9c1f_850c7ced7bd6.slice/crio-4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54 WatchSource:0}: Error finding container 4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54: Status 404 returned error can't find the container with id 4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54 Jul 15 08:45:01.033949 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.033909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" event={"ID":"8d696b34-a366-4051-ac36-d3b64807e467","Type":"ContainerStarted","Data":"4db84cb221be4cea585bcc9b0152a4ecd244b902ee9e6422749ad089544e6a30"} Jul 15 08:45:01.034156 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.033969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" event={"ID":"8d696b34-a366-4051-ac36-d3b64807e467","Type":"ContainerStarted","Data":"76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27"} Jul 15 08:45:01.035444 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.035415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" event={"ID":"f6eadcda-60f3-4aec-9c1f-850c7ced7bd6","Type":"ContainerStarted","Data":"f845150aa98ad40926f0c649f3e7c747e30d9d10aed1eeb3fb1e65db3f936566"} Jul 15 08:45:01.035583 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.035458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" event={"ID":"f6eadcda-60f3-4aec-9c1f-850c7ced7bd6","Type":"ContainerStarted","Data":"4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54"} Jul 15 08:45:01.036826 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.036798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" event={"ID":"5dce0f91-5618-4196-b403-cc15f22eb06f","Type":"ContainerStarted","Data":"d4967aaf4aad2fc56ec05791f658f50abe5ee55f7eefd5d2c63fd50e55072761"} Jul 15 08:45:01.036956 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.036840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" event={"ID":"5dce0f91-5618-4196-b403-cc15f22eb06f","Type":"ContainerStarted","Data":"8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4"} Jul 15 08:45:01.052213 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.052162 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" podStartSLOduration=1.052147837 podStartE2EDuration="1.052147837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:45:01.050118318 +0000 UTC m=+2678.469863122" watchObservedRunningTime="2026-07-15 08:45:01.052147837 +0000 UTC m=+2678.471892640" Jul 15 08:45:01.067902 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.067851 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" podStartSLOduration=1.067835634 podStartE2EDuration="1.067835634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:45:01.067157322 +0000 UTC m=+2678.486902124" watchObservedRunningTime="2026-07-15 08:45:01.067835634 +0000 UTC m=+2678.487580440" Jul 15 08:45:01.084320 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:01.084271 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" podStartSLOduration=1.084253958 podStartE2EDuration="1.084253958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:45:01.082800019 +0000 UTC m=+2678.502544846" watchObservedRunningTime="2026-07-15 08:45:01.084253958 +0000 UTC m=+2678.503998760" Jul 15 08:45:02.047384 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.047350 2565 generic.go:356] "Generic (PLEG): container finished" podID="8d696b34-a366-4051-ac36-d3b64807e467" containerID="4db84cb221be4cea585bcc9b0152a4ecd244b902ee9e6422749ad089544e6a30" exitCode=0 Jul 15 08:45:02.047820 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.047440 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" event={"ID":"8d696b34-a366-4051-ac36-d3b64807e467","Type":"ContainerDied","Data":"4db84cb221be4cea585bcc9b0152a4ecd244b902ee9e6422749ad089544e6a30"} Jul 15 08:45:02.048842 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.048817 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6eadcda-60f3-4aec-9c1f-850c7ced7bd6" containerID="f845150aa98ad40926f0c649f3e7c747e30d9d10aed1eeb3fb1e65db3f936566" exitCode=0 Jul 15 08:45:02.048913 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.048893 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" event={"ID":"f6eadcda-60f3-4aec-9c1f-850c7ced7bd6","Type":"ContainerDied","Data":"f845150aa98ad40926f0c649f3e7c747e30d9d10aed1eeb3fb1e65db3f936566"} Jul 15 08:45:02.050238 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.050216 2565 generic.go:356] "Generic (PLEG): container finished" podID="5dce0f91-5618-4196-b403-cc15f22eb06f" containerID="d4967aaf4aad2fc56ec05791f658f50abe5ee55f7eefd5d2c63fd50e55072761" exitCode=0 Jul 15 08:45:02.050351 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:02.050248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" event={"ID":"5dce0f91-5618-4196-b403-cc15f22eb06f","Type":"ContainerDied","Data":"d4967aaf4aad2fc56ec05791f658f50abe5ee55f7eefd5d2c63fd50e55072761"} Jul 15 08:45:03.220481 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:03.220451 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" Jul 15 08:45:03.273259 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:03.273230 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" Jul 15 08:45:03.276059 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:03.276038 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" Jul 15 08:45:04.062870 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.062836 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" Jul 15 08:45:04.063060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.062842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735085-mdtrc" event={"ID":"8d696b34-a366-4051-ac36-d3b64807e467","Type":"ContainerDied","Data":"76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27"} Jul 15 08:45:04.063060 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.063004 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c74d26332d25000bcb93eda0d97b9a6386f1084cff668bdc556d371e2b4e27" Jul 15 08:45:04.064146 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.064127 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" Jul 15 08:45:04.064268 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.064146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x" event={"ID":"f6eadcda-60f3-4aec-9c1f-850c7ced7bd6","Type":"ContainerDied","Data":"4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54"} Jul 15 08:45:04.064268 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.064187 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3ebd0a77b258cc001ad6370f20c2434c0d5c05d52e5c046f383d65a9290c54" Jul 15 08:45:04.065424 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.065381 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" event={"ID":"5dce0f91-5618-4196-b403-cc15f22eb06f","Type":"ContainerDied","Data":"8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4"} Jul 15 08:45:04.065496 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.065446 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b45f2d4d62706fb134c69d65e8fab7ebc247a611be76ebd439fcf54561d0ad4" Jul 15 08:45:04.065496 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:04.065388 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j" Jul 15 08:45:23.329497 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:23.329470 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:45:23.337852 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:23.337823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:45:23.345251 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:23.345230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:45:23.353038 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:45:23.353012 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m7n97_b4f8670d-87f7-4d3d-af8a-10a9afd44afe/ovn-acl-logging/0.log" Jul 15 08:48:23.506889 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:23.506849 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j"] Jul 15 08:48:23.514421 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:23.514373 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8e59d0-29735085-qxq5j"] Jul 15 08:48:25.116959 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:25.116923 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dce0f91-5618-4196-b403-cc15f22eb06f" path="/var/lib/kubelet/pods/5dce0f91-5618-4196-b403-cc15f22eb06f/volumes" Jul 15 08:48:38.840009 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:38.839976 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:48:38.840415 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:38.840230 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" containerID="cri-o://b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce" gracePeriod=30 Jul 15 08:48:43.294258 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:43.294220 2565 patch_prober.go:28] interesting pod/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 08:48:43.294642 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:43.294284 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 08:48:44.105894 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.105830 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:48:44.179954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.179919 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.179954 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.179962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180212 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.179990 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180212 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180212 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180307 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180349 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:48:44.180364 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xpxs\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs\") pod \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\" (UID: \"9e3420a8-13cc-4c2d-a930-243c3598c2d0\") " Jul 15 08:48:44.180719 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180575 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:44.180719 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180605 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:44.180955 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180917 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:44.180955 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180927 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-socket\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.181107 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180967 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istiod-ca-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.181107 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180984 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-credential-socket\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.181107 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.180997 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-workload-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.182588 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.182556 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:44.182701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.182617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 08:48:44.182701 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.182682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs" (OuterVolumeSpecName: "kube-api-access-9xpxs") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "kube-api-access-9xpxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:44.182884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.182864 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e3420a8-13cc-4c2d-a930-243c3598c2d0" (UID: "9e3420a8-13cc-4c2d-a930-243c3598c2d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 08:48:44.281726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.281700 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-envoy\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.281726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.281723 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-token\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.281726 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.281733 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xpxs\" (UniqueName: \"kubernetes.io/projected/9e3420a8-13cc-4c2d-a930-243c3598c2d0-kube-api-access-9xpxs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.281955 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.281742 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-data\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.281955 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.281750 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3420a8-13cc-4c2d-a930-243c3598c2d0-istio-podinfo\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:44.338891 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.338856 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerID="b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce" exitCode=0 Jul 15 08:48:44.339356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.338905 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" Jul 15 08:48:44.339356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.338954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerDied","Data":"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce"} Jul 15 08:48:44.339356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.339021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk" event={"ID":"9e3420a8-13cc-4c2d-a930-243c3598c2d0","Type":"ContainerDied","Data":"1af3b7fa34f8065928a9058d892cf3c4742122882d761563273be9932ed2d279"} Jul 15 08:48:44.339356 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.339057 2565 scope.go:117] "RemoveContainer" containerID="b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce" Jul 15 08:48:44.349375 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.349360 2565 scope.go:117] "RemoveContainer" containerID="0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa" Jul 15 08:48:44.357946 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.357923 2565 scope.go:117] "RemoveContainer" containerID="b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce" Jul 15 08:48:44.358198 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:48:44.358180 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce\": container with ID starting with b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce not found: ID does not exist" containerID="b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce" Jul 15 08:48:44.358277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.358212 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce"} err="failed to get container status \"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce\": rpc error: code = NotFound desc = could not find container \"b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce\": container with ID starting with b04f3f90b569ad0b6fe894b12a5d825451841271621b0bc3b665caac914aa7ce not found: ID does not exist" Jul 15 08:48:44.358277 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.358235 2565 scope.go:117] "RemoveContainer" containerID="0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa" Jul 15 08:48:44.358496 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:48:44.358479 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa\": container with ID starting with 0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa not found: ID does not exist" containerID="0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa" Jul 15 08:48:44.358544 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.358502 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa"} err="failed to get container status \"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa\": rpc error: code = NotFound desc = could not find container \"0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa\": container with ID starting with 0674211c6d17d136f05b48be01b70ff04e49e3b0f61a785268b76ee9ffeacdaa not found: ID does not exist" Jul 15 08:48:44.374576 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.374548 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:48:44.386839 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:44.386814 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8e59d0-openshift-default-78c865f847-5l9lk"] Jul 15 08:48:45.117664 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:45.117628 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" path="/var/lib/kubelet/pods/9e3420a8-13cc-4c2d-a930-243c3598c2d0/volumes" Jul 15 08:48:49.762157 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:49.762121 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:48:49.762609 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:49.762370 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" podUID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" containerName="maas-api" containerID="cri-o://7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2" gracePeriod=30 Jul 15 08:48:49.788287 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:49.788260 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x"] Jul 15 08:48:49.790776 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:49.790750 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1db148-29735085-vgm6x"] Jul 15 08:48:50.022346 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.022280 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:48:50.143520 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.143481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca\") pod \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " Jul 15 08:48:50.143695 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.143543 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7swp\" (UniqueName: \"kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp\") pod \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " Jul 15 08:48:50.143695 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.143610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") pod \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\" (UID: \"67e4ef85-923e-4aa9-84e8-98c6b470e9e9\") " Jul 15 08:48:50.143883 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.143860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67e4ef85-923e-4aa9-84e8-98c6b470e9e9" (UID: "67e4ef85-923e-4aa9-84e8-98c6b470e9e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 08:48:50.144001 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.143973 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:50.145632 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.145607 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67e4ef85-923e-4aa9-84e8-98c6b470e9e9" (UID: "67e4ef85-923e-4aa9-84e8-98c6b470e9e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 08:48:50.145714 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.145670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp" (OuterVolumeSpecName: "kube-api-access-l7swp") pod "67e4ef85-923e-4aa9-84e8-98c6b470e9e9" (UID: "67e4ef85-923e-4aa9-84e8-98c6b470e9e9"). InnerVolumeSpecName "kube-api-access-l7swp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 08:48:50.244958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.244919 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:50.244958 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.244947 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7swp\" (UniqueName: \"kubernetes.io/projected/67e4ef85-923e-4aa9-84e8-98c6b470e9e9-kube-api-access-l7swp\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\"" Jul 15 08:48:50.377516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.377423 2565 generic.go:356] "Generic (PLEG): container finished" podID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" containerID="7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2" exitCode=0 Jul 15 08:48:50.377516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.377464 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" Jul 15 08:48:50.377516 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.377505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" event={"ID":"67e4ef85-923e-4aa9-84e8-98c6b470e9e9","Type":"ContainerDied","Data":"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2"} Jul 15 08:48:50.377734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.377563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq" event={"ID":"67e4ef85-923e-4aa9-84e8-98c6b470e9e9","Type":"ContainerDied","Data":"88b4f6f07aa7ca89834f283714d2bb372eaca9ca0bb10d5b7960b8a55172808e"} Jul 15 08:48:50.377734 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.377587 2565 scope.go:117] "RemoveContainer" containerID="7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2" Jul 15 08:48:50.394863 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.394844 2565 scope.go:117] "RemoveContainer" containerID="7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2" Jul 15 08:48:50.395175 ip-10-0-139-41 kubenswrapper[2565]: E0715 08:48:50.395149 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2\": container with ID starting with 7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2 not found: ID does not exist" containerID="7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2" Jul 15 08:48:50.395273 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.395180 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2"} err="failed to get container status \"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2\": rpc error: code = NotFound desc = could not find container \"7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2\": container with ID starting with 7ed6ae537f70fe11d84a5cba9646bf3f39d1c12bfa330255888f036be972f3c2 not found: ID does not exist" Jul 15 08:48:50.404975 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.404952 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:48:50.408821 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:50.408801 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1db148-f5d8f456f-srcrq"] Jul 15 08:48:51.116095 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:51.116066 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" path="/var/lib/kubelet/pods/67e4ef85-923e-4aa9-84e8-98c6b470e9e9/volumes" Jul 15 08:48:51.116484 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:48:51.116353 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6eadcda-60f3-4aec-9c1f-850c7ced7bd6" path="/var/lib/kubelet/pods/f6eadcda-60f3-4aec-9c1f-850c7ced7bd6/volumes" Jul 15 08:49:11.356006 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:11.355970 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-599b49958f-6q29h_43489291-da25-4a70-9791-abf63f8c7c0a/manager/0.log" Jul 15 08:49:11.733638 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:11.733613 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-2g5cn_d49efb9a-8e4a-41a2-84e3-66e3b5329698/copy-manifests/0.log" Jul 15 08:49:11.740902 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:11.740873 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-2g5cn_d49efb9a-8e4a-41a2-84e3-66e3b5329698/manager/0.log" Jul 15 08:49:12.393084 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.393051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b5b9647fd-ncrnf_d3deee17-c3de-48f3-9487-097c0fb06aea/maas-api/0.log" Jul 15 08:49:12.514868 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.514837 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735055-n8rcm_5357d86a-d793-492b-b10e-7559572cc2cb/cleanup/0.log" Jul 15 08:49:12.640537 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.640503 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735070-nw2c8_209af120-e927-42aa-8e1e-74fddae0c24d/cleanup/0.log" Jul 15 08:49:12.757138 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.757118 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735085-mdtrc_8d696b34-a366-4051-ac36-d3b64807e467/cleanup/0.log" Jul 15 08:49:12.878549 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.878515 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-4dbdead5-66n9k_461bb60b-2fef-4140-919e-4d75a09304e2/revoke-keys/0.log" Jul 15 08:49:12.996318 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:12.996293 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f0ce8ae8-z9cpc_d03d3ba6-cf73-4887-aad7-ea267b6b2c6d/revoke-keys/0.log" Jul 15 08:49:13.116716 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.116626 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-ec03f1-5pbbf_837072e9-be6f-4867-823f-4388a766bab8/revoke-keys/0.log" Jul 15 08:49:13.235729 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.235693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-7a0e41-x5285_4ff0bb17-fd41-4277-9fb2-ea0d3bb3cf26/revoke-keys/0.log" Jul 15 08:49:13.360337 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.360304 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-93b7ed10-279x5_2c21d8d9-79e8-47d5-b135-556be5addd9e/revoke-keys/0.log" Jul 15 08:49:13.478184 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.478107 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-f6b673-5jjxt_714071ef-cab4-4d86-ac4a-280a3c74ee4d/revoke-keys/0.log" Jul 15 08:49:13.598627 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.598594 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-4c59e6-fzzhs_9b9dd329-ccc8-48c9-9868-16ac89623c72/revoke-keys/0.log" Jul 15 08:49:13.714189 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.714150 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-cdca1605-2xhnq_07e1d1a8-909c-4d3d-9e47-957385d10f4b/revoke-keys/0.log" Jul 15 08:49:13.834367 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.834341 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-13831f8c-5mgsj_59b20ef3-68e5-4c4d-94c0-cccb92b6db12/revoke-keys/0.log" Jul 15 08:49:13.952760 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:13.952726 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-2082f65c-qz4hj_31d3a6c1-1ab3-434a-9ebf-4c2bfe0a1f54/revoke-keys/0.log" Jul 15 08:49:14.319528 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:14.319497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vn5zb_a1f76b51-84dd-4c76-b3b4-a1461e5ce7fe/postgres/0.log" Jul 15 08:49:17.144428 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:17.144367 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m4622_b062e0e0-1066-43a8-853b-9de1b131d4d5/discovery/0.log" Jul 15 08:49:17.646248 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:17.646207 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-87ng7_a2059bfc-ed4a-4261-bad1-36996b060389/payload-pre-processing/0.log" Jul 15 08:49:17.770703 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:17.770676 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-blmdw_e58d70eb-1088-4c87-91d1-bd5a791b0aba/payload-processing/0.log" Jul 15 08:49:17.894109 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:17.894083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8844f56-rsmbf_55aeea10-575f-4ca6-b899-02f564240e75/router/0.log" Jul 15 08:49:18.376285 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:18.376258 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vz2k5_f7ff1151-115d-4607-bf0d-8fd1747b5126/main/0.log" Jul 15 08:49:18.494585 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:18.494552 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ggqqx_651da742-7e85-4eb1-8b44-b3fa5b4cdc86/main/0.log" Jul 15 08:49:26.078713 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:26.078679 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2s78_bbb9d64a-d1b4-4feb-a041-4de44d59beaa/global-pull-secret-syncer/0.log" Jul 15 08:49:26.262911 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:26.262883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhb8r_2b5fddbb-ff3c-4a37-9dbd-f57ff6b26ad9/konnectivity-agent/0.log" Jul 15 08:49:26.316533 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:26.316501 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-41.ec2.internal_b0a59e8f4957b010cc06fb503bb4835a/haproxy/0.log" Jul 15 08:49:33.333127 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.333100 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/alertmanager/0.log" Jul 15 08:49:33.360736 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.360711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/config-reloader/0.log" Jul 15 08:49:33.387344 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.387318 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/kube-rbac-proxy-web/0.log" Jul 15 08:49:33.416195 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.416176 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/kube-rbac-proxy/0.log" Jul 15 08:49:33.491180 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.491153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/kube-rbac-proxy-metric/0.log" Jul 15 08:49:33.539840 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.539770 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/prom-label-proxy/0.log" Jul 15 08:49:33.610207 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.610171 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff2026aa-196e-4b26-9068-25385c82330f/init-config-reloader/0.log" Jul 15 08:49:33.673784 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.673762 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-8vd7w_fb48dc08-fe83-496b-96ad-9889a57b224f/cluster-monitoring-operator/0.log" Jul 15 08:49:33.819735 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:33.819655 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-prvnq_f19970a3-05fb-4dde-afb0-888b40392d21/monitoring-plugin/0.log" Jul 15 08:49:34.029763 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.029716 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6qsr_6b82e419-9f91-4efb-bf58-83caf297304a/node-exporter/0.log" Jul 15 08:49:34.054332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.054293 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6qsr_6b82e419-9f91-4efb-bf58-83caf297304a/kube-rbac-proxy/0.log" Jul 15 08:49:34.078247 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.078184 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6qsr_6b82e419-9f91-4efb-bf58-83caf297304a/init-textfile/0.log" Jul 15 08:49:34.191332 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.191312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/prometheus/0.log" Jul 15 08:49:34.215421 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.215383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/config-reloader/0.log" Jul 15 08:49:34.238727 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.238698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/thanos-sidecar/0.log" Jul 15 08:49:34.266639 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.266618 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/kube-rbac-proxy-web/0.log" Jul 15 08:49:34.291319 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.291294 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/kube-rbac-proxy/0.log" Jul 15 08:49:34.318998 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.318966 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/kube-rbac-proxy-thanos/0.log" Jul 15 08:49:34.345080 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.345013 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68ce833d-f0db-4f19-946a-1880d5d0f09b/init-config-reloader/0.log" Jul 15 08:49:34.428208 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.428181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-c4q4b_bb053f9b-e5bf-422f-98ce-97ccaf2557cf/prometheus-operator-admission-webhook/0.log" Jul 15 08:49:34.461146 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.461118 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4495d98-c2hmw_384ecd2f-e689-489e-b7dc-64d52dbfe41e/telemeter-client/0.log" Jul 15 08:49:34.489232 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.489205 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4495d98-c2hmw_384ecd2f-e689-489e-b7dc-64d52dbfe41e/reload/0.log" Jul 15 08:49:34.513576 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.513554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4495d98-c2hmw_384ecd2f-e689-489e-b7dc-64d52dbfe41e/kube-rbac-proxy/0.log" Jul 15 08:49:34.546106 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.546083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/thanos-query/0.log" Jul 15 08:49:34.573137 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.573103 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/kube-rbac-proxy-web/0.log" Jul 15 08:49:34.602243 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.602162 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/kube-rbac-proxy/0.log" Jul 15 08:49:34.623609 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.623585 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/prom-label-proxy/0.log" Jul 15 08:49:34.650988 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.650964 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/kube-rbac-proxy-rules/0.log" Jul 15 08:49:34.675733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:34.675712 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d45c569c-wtk76_de593aa0-7112-4d96-ba10-5725750c3072/kube-rbac-proxy-metrics/0.log" Jul 15 08:49:35.598842 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.598814 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt"] Jul 15 08:49:35.599215 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599208 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599219 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599228 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599233 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599240 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6eadcda-60f3-4aec-9c1f-850c7ced7bd6" containerName="cleanup" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599245 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6eadcda-60f3-4aec-9c1f-850c7ced7bd6" containerName="cleanup" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599251 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d696b34-a366-4051-ac36-d3b64807e467" containerName="cleanup" Jul 15 08:49:35.599256 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599256 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d696b34-a366-4051-ac36-d3b64807e467" containerName="cleanup" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599274 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" containerName="maas-api" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599279 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" containerName="maas-api" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599285 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dce0f91-5618-4196-b403-cc15f22eb06f" containerName="cleanup" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599290 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dce0f91-5618-4196-b403-cc15f22eb06f" containerName="cleanup" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599347 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d696b34-a366-4051-ac36-d3b64807e467" containerName="cleanup" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599355 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6eadcda-60f3-4aec-9c1f-850c7ced7bd6" containerName="cleanup" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599361 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e3420a8-13cc-4c2d-a930-243c3598c2d0" containerName="istio-proxy" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599367 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e4ef85-923e-4aa9-84e8-98c6b470e9e9" containerName="maas-api" Jul 15 08:49:35.599479 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.599376 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dce0f91-5618-4196-b403-cc15f22eb06f" containerName="cleanup" Jul 15 08:49:35.602672 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.602655 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.604898 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.604881 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bt6dg\"/\"openshift-service-ca.crt\"" Jul 15 08:49:35.605008 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.604903 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bt6dg\"/\"kube-root-ca.crt\"" Jul 15 08:49:35.605540 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.605526 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bt6dg\"/\"default-dockercfg-nbpnw\"" Jul 15 08:49:35.610164 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.610145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt"] Jul 15 08:49:35.696499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.696473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-podres\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.696499 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.696504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-lib-modules\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.696708 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.696542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-proc\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.696708 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.696606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-sys\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.696708 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.696630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rxq\" (UniqueName: \"kubernetes.io/projected/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-kube-api-access-b6rxq\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797619 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-podres\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-lib-modules\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-proc\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797742 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-sys\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rxq\" (UniqueName: \"kubernetes.io/projected/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-kube-api-access-b6rxq\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-podres\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-proc\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-lib-modules\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.797884 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.797824 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-sys\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.806117 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.806095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rxq\" (UniqueName: \"kubernetes.io/projected/98ad3415-e4f7-4ac8-913f-b7a7460fe7d0-kube-api-access-b6rxq\") pod \"perf-node-gather-daemonset-n7xbt\" (UID: \"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0\") " pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:35.913466 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:35.913344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:36.051789 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.051760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt"] Jul 15 08:49:36.053304 ip-10-0-139-41 kubenswrapper[2565]: W0715 08:49:36.053271 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98ad3415_e4f7_4ac8_913f_b7a7460fe7d0.slice/crio-0767ee057a81229297702eeeb91966d1e41e107c0d84e554294026c22f93e2f8 WatchSource:0}: Error finding container 0767ee057a81229297702eeeb91966d1e41e107c0d84e554294026c22f93e2f8: Status 404 returned error can't find the container with id 0767ee057a81229297702eeeb91966d1e41e107c0d84e554294026c22f93e2f8 Jul 15 08:49:36.495756 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.495726 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/1.log" Jul 15 08:49:36.500352 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.500330 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r542h_34bf1e10-21d6-4ea5-8abe-30009956d7ce/console-operator/2.log" Jul 15 08:49:36.633702 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.633667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" event={"ID":"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0","Type":"ContainerStarted","Data":"76775f8fd53edb0a21a59b79f007649855aeba2ee5b168107a74d721e1d31227"} Jul 15 08:49:36.634102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.633721 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" event={"ID":"98ad3415-e4f7-4ac8-913f-b7a7460fe7d0","Type":"ContainerStarted","Data":"0767ee057a81229297702eeeb91966d1e41e107c0d84e554294026c22f93e2f8"} Jul 15 08:49:36.634102 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.633748 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:36.650040 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:36.649978 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" podStartSLOduration=1.649959035 podStartE2EDuration="1.649959035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 08:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 08:49:36.649624204 +0000 UTC m=+2954.069369043" watchObservedRunningTime="2026-07-15 08:49:36.649959035 +0000 UTC m=+2954.069703837" Jul 15 08:49:37.661794 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:37.661758 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-5kfn6_7f3d67b1-ae4c-4494-b8be-54edcc5977c9/volume-data-source-validator/0.log" Jul 15 08:49:38.700867 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:38.700839 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-896pl_55f885a2-8fd2-4059-a3a5-988292116572/dns/0.log" Jul 15 08:49:38.723986 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:38.723965 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-896pl_55f885a2-8fd2-4059-a3a5-988292116572/kube-rbac-proxy/0.log" Jul 15 08:49:38.848551 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:38.848524 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jhvd_a2d9b9a1-0ca0-49b9-ae9d-d5dc4410a2c6/dns-node-resolver/0.log" Jul 15 08:49:39.535862 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:39.535821 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmdh8_1d4cff77-0eaf-4210-9839-c8a3cbe90021/node-ca/0.log" Jul 15 08:49:40.645673 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:40.645640 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m4622_b062e0e0-1066-43a8-853b-9de1b131d4d5/discovery/0.log" Jul 15 08:49:40.819035 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:40.818982 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-87ng7_a2059bfc-ed4a-4261-bad1-36996b060389/payload-pre-processing/0.log" Jul 15 08:49:40.851171 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:40.851134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-blmdw_e58d70eb-1088-4c87-91d1-bd5a791b0aba/payload-processing/0.log" Jul 15 08:49:40.874733 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:40.874705 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8844f56-rsmbf_55aeea10-575f-4ca6-b899-02f564240e75/router/0.log" Jul 15 08:49:41.606052 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:41.606016 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8s45p_6a0b8373-62f9-4798-a412-b5ba19e87b42/serve-healthcheck-canary/0.log" Jul 15 08:49:42.124977 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.124945 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-sf7fl_a0fe38cf-6ff3-4965-8bdf-2472b4abb40b/insights-operator/0.log" Jul 15 08:49:42.126130 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.126101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-sf7fl_a0fe38cf-6ff3-4965-8bdf-2472b4abb40b/insights-operator/1.log" Jul 15 08:49:42.147475 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.147452 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fcm7_944e658e-f32c-449b-9be1-1f7d5e922c18/kube-rbac-proxy/0.log" Jul 15 08:49:42.170460 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.170440 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fcm7_944e658e-f32c-449b-9be1-1f7d5e922c18/exporter/0.log" Jul 15 08:49:42.196666 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.196636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fcm7_944e658e-f32c-449b-9be1-1f7d5e922c18/extractor/0.log" Jul 15 08:49:42.648522 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:42.648496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bt6dg/perf-node-gather-daemonset-n7xbt" Jul 15 08:49:44.958621 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:44.958594 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-599b49958f-6q29h_43489291-da25-4a70-9791-abf63f8c7c0a/manager/0.log" Jul 15 08:49:45.113931 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:45.113903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-2g5cn_d49efb9a-8e4a-41a2-84e3-66e3b5329698/manager/0.log" Jul 15 08:49:45.137579 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:45.137549 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b8c85cff6-2g5cn_d49efb9a-8e4a-41a2-84e3-66e3b5329698/copy-manifests/0.log" Jul 15 08:49:46.555481 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:46.555453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bdbd9fc98-mg24w_08f26b44-40d4-4682-b10c-c6901fb418a6/manager/0.log" Jul 15 08:49:46.774631 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:46.774594 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m4622_b062e0e0-1066-43a8-853b-9de1b131d4d5/discovery/0.log" Jul 15 08:49:46.905685 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:46.905607 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-87ng7_a2059bfc-ed4a-4261-bad1-36996b060389/payload-pre-processing/0.log" Jul 15 08:49:46.925655 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:46.925627 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-blmdw_e58d70eb-1088-4c87-91d1-bd5a791b0aba/payload-processing/0.log" Jul 15 08:49:46.944051 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:46.944030 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8844f56-rsmbf_55aeea10-575f-4ca6-b899-02f564240e75/router/0.log" Jul 15 08:49:48.219570 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:48.219543 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m4622_b062e0e0-1066-43a8-853b-9de1b131d4d5/discovery/0.log" Jul 15 08:49:48.357092 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:48.357066 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-87ng7_a2059bfc-ed4a-4261-bad1-36996b060389/payload-pre-processing/0.log" Jul 15 08:49:48.382122 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:48.382093 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-blmdw_e58d70eb-1088-4c87-91d1-bd5a791b0aba/payload-processing/0.log" Jul 15 08:49:48.397630 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:48.397611 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8844f56-rsmbf_55aeea10-575f-4ca6-b899-02f564240e75/router/0.log" Jul 15 08:49:52.049974 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:52.049943 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-4gsng_f8d7f2ca-b85d-4084-9138-4d71c041be8a/kube-storage-version-migrator-operator/1.log" Jul 15 08:49:52.050772 ip-10-0-139-41 kubenswrapper[2565]: I0715 08:49:52.050751 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-4gsng_f8d7f2ca-b85d-4084-9138-4d71c041be8a/kube-storage-version-migrator-operator/0.log"