Jul 17 21:06:35.204523 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 21:06:35.204534 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 21:06:35.204544 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 21:06:35.204883 ip-10-0-129-137 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 21:06:45.288546 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 21:06:45.288562 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 85ebfc6ba2c04e52a8f36f6fc4b3e0db -- Jul 17 21:08:45.110759 ip-10-0-129-137 systemd[1]: Starting Kubernetes Kubelet... Jul 17 21:08:45.501687 ip-10-0-129-137 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:08:45.501687 ip-10-0-129-137 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 21:08:45.501687 ip-10-0-129-137 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:08:45.501687 ip-10-0-129-137 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 21:08:45.501687 ip-10-0-129-137 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:08:45.503878 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.503799 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 21:08:45.511016 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.510994 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.511016 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511012 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.511016 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511017 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.511016 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511020 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.511016 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511023 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511026 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511029 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511034 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511037 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511040 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511043 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511045 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511048 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511051 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511054 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511056 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511059 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511062 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511064 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511067 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511070 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511072 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511075 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.511208 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511077 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511080 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511082 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511085 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511087 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511090 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511092 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511095 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511099 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511101 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511104 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511106 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511109 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511111 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511114 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511117 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511120 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511122 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511125 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511128 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.511677 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511130 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511133 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511137 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511139 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511142 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511144 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511147 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511150 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511153 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511155 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511158 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511161 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511163 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511166 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511168 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511170 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511173 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511176 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511178 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.512190 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511180 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511184 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511189 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511192 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511195 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511198 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511201 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511204 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511207 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511209 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511212 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511215 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511217 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511222 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511224 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511227 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511230 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511232 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511235 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.512669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511237 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511240 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511243 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511245 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511247 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511250 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511253 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511255 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511257 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511260 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511262 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511266 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511269 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511272 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511275 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511278 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511281 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.511284 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512025 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.513128 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512033 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512038 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512043 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512047 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512051 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512054 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512057 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512060 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512062 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512065 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512068 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512071 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512073 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512076 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512079 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512082 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512085 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512087 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512090 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.513582 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512093 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512095 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512098 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512101 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512103 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512107 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512109 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512112 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512114 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512116 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512120 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512122 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512125 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512128 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512130 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512133 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512135 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512138 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512141 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.514041 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512143 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512146 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512149 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512152 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512154 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512157 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512159 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512162 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512164 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512167 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512169 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512179 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512182 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512184 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512187 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512189 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512192 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512194 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512197 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512199 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.514517 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512202 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512205 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512207 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512210 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512212 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512215 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512217 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512220 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512223 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512225 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512228 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512230 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512233 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512235 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512238 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512241 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512243 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512246 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512248 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512251 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512253 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.514986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512256 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512258 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512261 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512263 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512266 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512268 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512271 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512273 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512276 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512279 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512282 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512285 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512288 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512290 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512293 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512296 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512299 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512302 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512304 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512679 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.515475 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512684 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512687 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512689 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512692 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512695 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512698 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512700 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512703 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512706 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512708 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512711 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512713 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512715 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512718 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512720 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512723 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512726 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512728 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512730 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.515968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512733 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512736 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512738 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512740 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512743 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512745 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512747 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512750 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512753 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512755 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512757 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512760 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512763 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512765 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512768 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512770 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512773 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512775 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512779 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512782 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.516434 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512785 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512787 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512790 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512794 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512798 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512800 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512803 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512805 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512808 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512811 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512813 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512816 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512818 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512820 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512822 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512825 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512827 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512830 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512832 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512837 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.516952 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512839 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512842 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512844 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512846 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512849 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512851 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512854 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512856 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512859 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512861 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512864 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512867 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512869 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512872 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512875 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512877 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512879 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512882 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512884 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512888 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.517437 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512891 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512894 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512897 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512900 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512902 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512905 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512908 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512910 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512912 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512916 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512918 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512922 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512924 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512927 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512929 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512932 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512934 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512937 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.512939 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513011 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 21:08:45.517933 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513018 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513024 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513030 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513042 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513047 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513054 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513059 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513062 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513065 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513068 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513071 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513074 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513077 2573 flags.go:64] FLAG: --cgroup-root="" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513079 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513083 2573 flags.go:64] FLAG: --client-ca-file="" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513095 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513098 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513102 2573 flags.go:64] FLAG: --cluster-domain="" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513105 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513108 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513111 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513114 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513118 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 21:08:45.518407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513123 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513126 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513129 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513132 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513135 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513138 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513141 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513144 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513148 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513151 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513154 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513157 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513159 2573 flags.go:64] FLAG: --enable-server="true" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513162 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513166 2573 flags.go:64] FLAG: --event-burst="100" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513169 2573 flags.go:64] FLAG: --event-qps="50" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513172 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513175 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513178 2573 flags.go:64] FLAG: --eviction-hard="" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513181 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513184 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513187 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513190 2573 flags.go:64] FLAG: --eviction-soft="" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513193 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513195 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 21:08:45.518960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513199 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513202 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513204 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513207 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513210 2573 flags.go:64] FLAG: --feature-gates="" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513213 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513216 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513219 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513222 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513225 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513228 2573 flags.go:64] FLAG: --help="false" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513231 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513234 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513237 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513240 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513243 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513246 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513249 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513251 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513254 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513257 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513260 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513262 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513265 2573 flags.go:64] FLAG: --kube-reserved="" Jul 17 21:08:45.519567 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513268 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513270 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513273 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513276 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513278 2573 flags.go:64] FLAG: --lock-file="" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513281 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513283 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513286 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513291 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513295 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513297 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513300 2573 flags.go:64] FLAG: --logging-format="text" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513303 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513306 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513308 2573 flags.go:64] FLAG: --manifest-url="" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513311 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513315 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513318 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513322 2573 flags.go:64] FLAG: --max-pods="110" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513324 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513327 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513330 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513333 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513335 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513338 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 21:08:45.520173 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513341 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513349 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513352 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513355 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513358 2573 flags.go:64] FLAG: --pod-cidr="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513361 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513366 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513369 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513372 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513375 2573 flags.go:64] FLAG: --port="10250" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513378 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513380 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-012341ac6e6b0ac3a" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513383 2573 flags.go:64] FLAG: --qos-reserved="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513386 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513389 2573 flags.go:64] FLAG: --register-node="true" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513391 2573 flags.go:64] FLAG: --register-with-taints="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513395 2573 flags.go:64] FLAG: --registry-burst="10" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513398 2573 flags.go:64] FLAG: --registry-qps="5" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513401 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513403 2573 flags.go:64] FLAG: --reserved-memory="" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513407 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513410 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513413 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513416 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513418 2573 flags.go:64] FLAG: --runonce="false" Jul 17 21:08:45.520862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513421 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513424 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513427 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513430 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513432 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513435 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513438 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513440 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513446 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513448 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513451 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513454 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513457 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513460 2573 flags.go:64] FLAG: --system-cgroups="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513462 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513467 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513470 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513472 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513478 2573 flags.go:64] FLAG: --tls-min-version="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513480 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513483 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513500 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513503 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513506 2573 flags.go:64] FLAG: --v="2" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513509 2573 flags.go:64] FLAG: --version="false" Jul 17 21:08:45.521479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513514 2573 flags.go:64] FLAG: --vmodule="" Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513517 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.513520 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513641 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513645 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513648 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513651 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513654 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513657 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513661 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513665 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513669 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513672 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513674 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513677 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513681 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513684 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513687 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513689 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.522085 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513693 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513695 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513698 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513701 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513703 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513706 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513708 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513711 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513713 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513716 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513718 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513721 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513723 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513727 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513731 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513733 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513736 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513738 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513741 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.522538 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513743 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513746 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513749 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513751 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513754 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513757 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513759 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513761 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513764 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513768 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513770 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513773 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513775 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513777 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513780 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513783 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513785 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513787 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513790 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513792 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.522990 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513795 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513797 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513799 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513802 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513804 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513806 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513810 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513813 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513816 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513818 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513821 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513823 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513825 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513828 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513831 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513833 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513835 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513838 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513840 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513843 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.523478 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513845 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513849 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513852 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513854 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513856 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513859 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513861 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513864 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513866 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513868 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513871 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513873 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513875 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513878 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513882 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513885 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513887 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513890 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513894 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513897 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.523968 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513900 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513903 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513906 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.513908 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.514909 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523130 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523146 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523165 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523174 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523225 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523231 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523234 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523237 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523240 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.524445 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523243 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523246 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523249 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523252 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523254 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523257 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523259 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523261 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523264 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523266 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523269 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523271 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523274 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523276 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523278 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523281 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523283 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523285 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523288 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523290 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.524916 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523292 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523295 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523297 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523299 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523304 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523307 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523309 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523312 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523314 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523318 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523321 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523325 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523329 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523332 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523334 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523337 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523339 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523342 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523344 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.525406 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523347 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523349 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523352 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523354 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523356 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523359 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523362 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523365 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523369 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523372 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523374 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523377 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523380 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523382 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523385 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523388 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523390 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523393 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523396 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523399 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.525911 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523401 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523404 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523406 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523408 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523411 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523413 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523416 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523418 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523421 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523423 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523425 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523428 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523430 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523433 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523435 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523437 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523440 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523442 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523445 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523447 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.526382 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523450 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523452 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523455 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523457 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523460 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523463 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523465 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523468 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523470 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523472 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523475 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523477 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523480 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523499 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523504 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523511 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:08:45.526876 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523622 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523628 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523633 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523636 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523639 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523641 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523644 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523646 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523648 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523651 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523653 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523656 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523658 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523661 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523663 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523666 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523668 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523671 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523673 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:08:45.527259 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523676 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523679 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523681 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523685 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523688 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523691 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523693 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523697 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523700 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523704 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523706 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523709 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523711 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523714 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523716 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523719 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523721 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523723 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523726 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:08:45.527726 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523728 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523731 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523733 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523735 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523738 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523740 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523743 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523745 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523748 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523750 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523752 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523755 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523757 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523759 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523762 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523764 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523767 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523769 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523771 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523774 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:08:45.528166 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523776 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523778 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523781 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523784 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523786 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523790 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523793 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523795 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523797 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523799 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523802 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523805 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523807 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523809 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523812 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523814 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523816 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523819 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523821 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523823 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:08:45.528644 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523826 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523829 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523831 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523833 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523835 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523838 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523840 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523843 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523845 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523847 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523849 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523852 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523854 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523856 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523859 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523861 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523864 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523868 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523870 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:08:45.529155 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523873 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:45.523875 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523880 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.523977 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.526302 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.528178 2573 server.go:1019] "Starting client certificate rotation" Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.528277 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:08:45.529621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.528310 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:08:45.531620 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.531606 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:08:45.533143 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.533116 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:08:45.547324 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.547306 2573 log.go:25] "Validated CRI v1 runtime API" Jul 17 21:08:45.549306 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.549290 2573 log.go:25] "Validated CRI v1 image API" Jul 17 21:08:45.550091 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.550073 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 21:08:45.554707 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.554686 2573 fs.go:135] Filesystem UUIDs: map[3467e564-3d4a-4ad8-99ab-713549483ddb:/dev/nvme0n1p4 6ced37d3-6345-4a19-ae45-9c92373b3194:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 17 21:08:45.554786 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.554706 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 21:08:45.559014 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.558993 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:08:45.560867 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.560764 2573 manager.go:216] Machine: {Timestamp:2026-07-17 21:08:45.559381344 +0000 UTC m=+0.333664541 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110515 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec224c9522c52e1723a156e2943491f2 SystemUUID:ec224c95-22c5-2e17-23a1-56e2943491f2 BootID:85ebfc6b-a2c0-4e52-a8f3-6f6fc4b3e0db Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:bc:52:bc:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:bc:52:bc:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:04:17:df:84:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 21:08:45.560867 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.560862 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 21:08:45.560965 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.560932 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 21:08:45.564211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.564190 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 21:08:45.564350 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.564214 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 21:08:45.565124 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.565113 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 21:08:45.565155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.565126 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 21:08:45.565155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.565138 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:08:45.565840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.565831 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:08:45.567820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.567810 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 21:08:45.571772 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571761 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:08:45.571872 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571863 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 21:08:45.571909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571893 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 17 21:08:45.571909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571908 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 21:08:45.571983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571921 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 21:08:45.571983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571931 2573 kubelet.go:398] "Adding apiserver pod source" Jul 17 21:08:45.571983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.571940 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 21:08:45.572259 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.572248 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:08:45.572291 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.572267 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:08:45.575094 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.575074 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 21:08:45.577063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.577048 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 21:08:45.577147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.577093 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 21:08:45.578337 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578324 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578343 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578352 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578359 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578371 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578378 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578390 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578398 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578415 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 21:08:45.578421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578424 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 21:08:45.578697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578440 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 21:08:45.578697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.578449 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 21:08:45.579175 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.579162 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 21:08:45.579222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.579178 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 21:08:45.581193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.581174 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 21:08:45.581274 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.581223 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 21:08:45.581274 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.581237 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 21:08:45.582185 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.582172 2573 server.go:1268] "Started kubelet" Jul 17 21:08:45.582325 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.582262 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 21:08:45.582388 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.582290 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 21:08:45.587416 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.587390 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 21:08:45.587869 ip-10-0-129-137 systemd[1]: Started Kubernetes Kubelet. Jul 17 21:08:45.588122 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.588099 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jg2wc" Jul 17 21:08:45.590651 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.589533 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 21:08:45.590651 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.589664 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 21:08:45.590651 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.590195 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 17 21:08:45.593200 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.592406 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18c32fe21e0d3c4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-17 21:08:45.582146638 +0000 UTC m=+0.356429834,LastTimestamp:2026-07-17 21:08:45.582146638 +0000 UTC m=+0.356429834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}" Jul 17 21:08:45.595422 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.595406 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 21:08:45.595422 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.595413 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 21:08:45.595548 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.595520 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:45.596038 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596017 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 21:08:45.596038 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596036 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 21:08:45.596334 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596117 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 17 21:08:45.596334 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596127 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 17 21:08:45.596334 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596247 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 21:08:45.596472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596432 2573 factory.go:55] Registering systemd factory Jul 17 21:08:45.596472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596459 2573 factory.go:223] Registration of the systemd container factory successfully Jul 17 21:08:45.596684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596668 2573 factory.go:153] Registering CRI-O factory Jul 17 21:08:45.596684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596682 2573 factory.go:223] Registration of the crio container factory successfully Jul 17 21:08:45.596795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596722 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 21:08:45.596795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596738 2573 factory.go:103] Registering Raw factory Jul 17 21:08:45.596795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.596748 2573 manager.go:1196] Started watching for new ooms in manager Jul 17 21:08:45.597500 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.597470 2573 manager.go:349] Starting recovery of all containers Jul 17 21:08:45.599075 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.599052 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jg2wc" Jul 17 21:08:45.608439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.608308 2573 manager.go:354] Recovery completed Jul 17 21:08:45.608541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.608443 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:45.611019 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.610995 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.612978 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.612967 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.621830 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.621814 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.621917 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.621841 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.621917 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.621851 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.622369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.622352 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 21:08:45.622369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.622367 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 21:08:45.622474 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.622381 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.622474 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.622383 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:08:45.623084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623064 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.623157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623091 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.623157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623103 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.623578 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623564 2573 policy_none.go:49] "None policy: Start" Jul 17 21:08:45.623631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623580 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 21:08:45.623631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.623590 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 21:08:45.624624 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.624613 2573 policy_none.go:47] "Start" Jul 17 21:08:45.659402 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.659381 2573 manager.go:337] "Starting Device Plugin manager" Jul 17 21:08:45.659522 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.659443 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 21:08:45.659522 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.659456 2573 server.go:87] "Starting device plugin registration server" Jul 17 21:08:45.659771 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.659739 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 21:08:45.659843 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.659775 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 21:08:45.660761 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.660067 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 21:08:45.660761 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.660164 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 21:08:45.660761 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.660172 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 21:08:45.661376 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.661354 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 21:08:45.661728 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.661711 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:45.760018 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.759976 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.761099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.760821 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.761099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.760852 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.761099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.760867 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.761099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.760893 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.761333 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.761305 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 21:08:45.762642 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.762625 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 21:08:45.762723 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.762659 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 21:08:45.762723 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.762678 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 21:08:45.762723 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.762706 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 21:08:45.764287 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.764271 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:45.767729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.767715 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.767798 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.767735 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-137.ec2.internal\": node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:45.833904 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.833880 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:45.863729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.863701 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"] Jul 17 21:08:45.863820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.863759 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.864652 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.864638 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.864717 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.864665 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.864717 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.864675 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.865867 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.865855 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.866002 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.865988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.866071 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866039 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.866480 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866467 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.866553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866512 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.866553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866516 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.866553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866540 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.866553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866547 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.866553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.866553 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.867757 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.867743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.867810 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.867773 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:08:45.868406 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.868393 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:08:45.868470 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.868418 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:08:45.868470 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.868427 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:08:45.887025 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.887004 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.890710 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.890696 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.897939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.897924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.897992 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.897947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.897992 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.897965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.934070 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:45.934051 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:45.998789 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.998853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.998853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.998920 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998864 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.998920 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:45.998981 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:45.998919 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:46.035040 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.035013 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.135625 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.135591 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.190777 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.190754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:46.194384 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.194316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:46.235917 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.235892 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.336424 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.336371 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.436826 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.436804 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.528338 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.528310 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 21:08:46.528853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.528449 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:08:46.528853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.528478 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:08:46.537600 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.537580 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.604040 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.603966 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:03:47 +0000 UTC" deadline="2028-04-20 15:25:01.971020102 +0000 UTC" Jul 17 21:08:46.604040 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.603996 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15426h16m15.367028554s" Jul 17 21:08:46.630618 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.630590 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:46.637858 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.637838 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.662218 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:46.662184 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf3a63f5e16fba7955768004cdb0f7c.slice/crio-e29e3b1639b83dcba2e2265b22a1801e4a7ad253fff4aa9169faecda8d16958b WatchSource:0}: Error finding container e29e3b1639b83dcba2e2265b22a1801e4a7ad253fff4aa9169faecda8d16958b: Status 404 returned error can't find the container with id e29e3b1639b83dcba2e2265b22a1801e4a7ad253fff4aa9169faecda8d16958b Jul 17 21:08:46.667031 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.667006 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:08:46.738133 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.738114 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.766472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.766427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerStarted","Data":"e29e3b1639b83dcba2e2265b22a1801e4a7ad253fff4aa9169faecda8d16958b"} Jul 17 21:08:46.838873 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:46.838854 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 17 21:08:46.909023 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:46.909001 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:46.909239 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:46.909221 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf449d6f92faa6a51a14c54944141985e.slice/crio-971eb9a032cc85c02beea17f450a0c1ebab19286fe5269793abeef1170d1c318 WatchSource:0}: Error finding container 971eb9a032cc85c02beea17f450a0c1ebab19286fe5269793abeef1170d1c318: Status 404 returned error can't find the container with id 971eb9a032cc85c02beea17f450a0c1ebab19286fe5269793abeef1170d1c318 Jul 17 21:08:48.465150 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:48.465129 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 17 21:08:48.472406 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:48.472391 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:08:48.473307 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:48.473296 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 17 21:08:48.484973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:48.484956 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:08:48.890904 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:48.890861 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:49.042576 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.042548 2573 apiserver.go:52] "Watching apiserver" Jul 17 21:08:49.052106 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.052081 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 21:08:49.054282 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.054254 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k2hd6","kube-system/konnectivity-agent-r8m8k","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q48tf","openshift-image-registry/node-ca-cp4k9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-7cg54","openshift-multus/multus-dpbg6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v","openshift-dns/node-resolver-p4ths","openshift-multus/network-metrics-daemon-hnhvt","openshift-network-diagnostics/network-check-target-w929x","openshift-network-operator/iptables-alerter-bttgg","openshift-ovn-kubernetes/ovnkube-node-vb84k"] Jul 17 21:08:49.055692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.055673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.057330 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.057305 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.058537 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.058517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.059153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.059134 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.059569 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.059550 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.061211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061191 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 21:08:49.061488 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061467 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 21:08:49.061669 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061650 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 21:08:49.061739 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061719 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.061811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\"" Jul 17 21:08:49.061914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.061897 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.062224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.062203 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 21:08:49.062343 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.062328 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\"" Jul 17 21:08:49.064825 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.064804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.065391 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.065374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.067812 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.067792 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 21:08:49.068980 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.068876 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.069116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069098 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 21:08:49.069349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069303 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\"" Jul 17 21:08:49.069349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069324 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.069573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069556 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.069712 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069100 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 21:08:49.069764 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069712 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 21:08:49.069922 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.069888 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 21:08:49.070073 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.070058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.070297 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.070274 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.070529 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.070510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\"" Jul 17 21:08:49.071607 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.071589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.073826 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.073804 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.074123 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.074103 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\"" Jul 17 21:08:49.074353 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.074336 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 21:08:49.074762 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.074744 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 21:08:49.075133 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075112 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 21:08:49.075205 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075178 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.075257 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.075354 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075338 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.075479 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.075334 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:49.075542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075209 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\"" Jul 17 21:08:49.075614 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\"" Jul 17 21:08:49.075679 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075664 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.075850 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-tmp\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.075918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cnibin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.075918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cni-binary-copy\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.075918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-daemon-config\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mvj\" (UniqueName: \"kubernetes.io/projected/cfda85cb-e637-42da-a583-f28076b66cd2-kube-api-access-c9mvj\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.076064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2923af86-87e3-42e0-b039-7a0643df290b-ovn-node-metrics-cert\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.075996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v9b4\" (UniqueName: \"kubernetes.io/projected/2923af86-87e3-42e0-b039-7a0643df290b-kube-api-access-2v9b4\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-multus-certs\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-systemd-units\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-system-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076398 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-multus\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076398 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-hostroot\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-conf-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-netns\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-script-lib\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-systemd\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.076673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-netns\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.076673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-host-slash\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.076673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.076673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-etc-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-bin\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-env-overrides\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-log-socket\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-modprobe-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-run\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-var-lib-kubelet\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.076909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2nz\" (UniqueName: \"kubernetes.io/projected/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-kube-api-access-hp2nz\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-host\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.076979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d39a33-6c3a-4dfb-a70e-9714ccc04802-host\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-kubelet\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-cnibin\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-ovn\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-bin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.077154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-netd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-node-log\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysconfig\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d39a33-6c3a-4dfb-a70e-9714ccc04802-serviceca\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-k8s-cni-cncf-io\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-os-release\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.077519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.077863 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.077848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.078300 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-slash\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.078384 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.078440 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-iptables-alerter-script\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.078440 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-conf\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.078530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-sys\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.078530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-socket-dir-parent\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.078530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-kubelet\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.078680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078527 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-tuned\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.078680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4f55\" (UniqueName: \"kubernetes.io/projected/08d39a33-6c3a-4dfb-a70e-9714ccc04802-kube-api-access-f4f55\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.078680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.078827 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-etc-kubernetes\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.078827 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd99\" (UniqueName: \"kubernetes.io/projected/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-kube-api-access-2dd99\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.078827 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.078960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.078960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.079169 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.079150 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 21:08:49.079244 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.079225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\"" Jul 17 21:08:49.079295 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.079250 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 21:08:49.079633 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.079612 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.079715 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.079670 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:49.081026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.080436 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 21:08:49.081026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.080554 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:49.081026 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.080617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:08:49.081026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.080728 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:08:49.081026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.080836 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 21:08:49.081382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081111 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\"" Jul 17 21:08:49.081382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.078951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-systemd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.081473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.081473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-config\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.081473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-kubernetes\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.081625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czqt\" (UniqueName: \"kubernetes.io/projected/15105daa-459e-46fd-aab4-c51a3a0ea806-kube-api-access-7czqt\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.081625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081508 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-var-lib-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.081625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-os-release\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.081625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.081551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-lib-modules\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.101677 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.101619 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfxq9" Jul 17 21:08:49.108575 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.108550 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfxq9" Jul 17 21:08:49.167266 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.167229 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 21:08:49.182572 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-modprobe-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-run\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-var-lib-kubelet\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.182708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2nz\" (UniqueName: \"kubernetes.io/projected/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-kube-api-access-hp2nz\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.182708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-host\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4js\" (UniqueName: \"kubernetes.io/projected/56fe822b-b110-43b8-bc35-b34a75fef2dd-kube-api-access-tg4js\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-kubelet-config\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-dbus\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3173e73-266f-4600-a98b-97d2c3a33af7-agent-certs\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d39a33-6c3a-4dfb-a70e-9714ccc04802-host\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-kubelet\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-cnibin\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-ovn\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-modprobe-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07f27250-1057-446a-99a5-e49eb4eb1354-hosts-file\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-run\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.182987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f27250-1057-446a-99a5-e49eb4eb1354-tmp-dir\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-bin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-netd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3173e73-266f-4600-a98b-97d2c3a33af7-konnectivity-ca\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-node-log\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysconfig\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d39a33-6c3a-4dfb-a70e-9714ccc04802-serviceca\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-k8s-cni-cncf-io\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-os-release\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-host\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-slash\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-iptables-alerter-script\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.183594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-conf\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-sys\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-socket-dir-parent\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-kubelet\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-tuned\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4f55\" (UniqueName: \"kubernetes.io/projected/08d39a33-6c3a-4dfb-a70e-9714ccc04802-kube-api-access-f4f55\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-etc-kubernetes\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd99\" (UniqueName: \"kubernetes.io/projected/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-kube-api-access-2dd99\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-systemd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-config\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-kubernetes\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czqt\" (UniqueName: \"kubernetes.io/projected/15105daa-459e-46fd-aab4-c51a3a0ea806-kube-api-access-7czqt\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.184392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjst6\" (UniqueName: \"kubernetes.io/projected/20ef74bf-c023-43cb-90f1-b8db835926b4-kube-api-access-cjst6\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-var-lib-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.182815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-var-lib-kubelet\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-os-release\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183969 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d39a33-6c3a-4dfb-a70e-9714ccc04802-host\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.183991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-lib-modules\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-kubelet\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-tmp\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-cnibin\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-ovn\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cnibin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cni-binary-copy\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-daemon-config\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184158 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-d\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mvj\" (UniqueName: \"kubernetes.io/projected/cfda85cb-e637-42da-a583-f28076b66cd2-kube-api-access-c9mvj\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2923af86-87e3-42e0-b039-7a0643df290b-ovn-node-metrics-cert\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v9b4\" (UniqueName: \"kubernetes.io/projected/2923af86-87e3-42e0-b039-7a0643df290b-kube-api-access-2v9b4\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-multus-certs\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-systemd-units\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-bin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-system-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-multus\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-node-log\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184485 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-var-lib-cni-multus\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-hostroot\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-conf-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.185729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-netns\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-script-lib\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-systemd\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-netns\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7d7\" (UniqueName: \"kubernetes.io/projected/07f27250-1057-446a-99a5-e49eb4eb1354-kube-api-access-hr7d7\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-daemon-config\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-host-slash\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184730 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-host-slash\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-etc-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-etc-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184833 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-bin\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184844 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysconfig\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-k8s-cni-cncf-io\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-os-release\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185029 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185080 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-system-cni-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfda85cb-e637-42da-a583-f28076b66cd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.186547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185572 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfda85cb-e637-42da-a583-f28076b66cd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-slash\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-systemd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-iptables-alerter-script\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-sysctl-conf\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-sys\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.185989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-socket-dir-parent\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-kubelet\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-config\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186325 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-kubernetes\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-multus-certs\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-systemd-units\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-run-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-system-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-etc-kubernetes\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.187439 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186599 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-var-lib-openvswitch\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-netd\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-os-release\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186681 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cnibin\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.186829 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-lib-modules\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187115 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-cni-binary-copy\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.184793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-cni-bin\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-env-overrides\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-log-socket\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187262 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-log-socket\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187509 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-cni-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-hostroot\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-multus-conf-dir\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d39a33-6c3a-4dfb-a70e-9714ccc04802-serviceca\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-host-run-netns\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-systemd\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2923af86-87e3-42e0-b039-7a0643df290b-host-run-netns\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.187997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-env-overrides\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.188363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.188146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2923af86-87e3-42e0-b039-7a0643df290b-ovnkube-script-lib\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.189310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.189271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-etc-tuned\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.189398 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.189306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15105daa-459e-46fd-aab4-c51a3a0ea806-tmp\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.193160 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.193093 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd99\" (UniqueName: \"kubernetes.io/projected/bbeba6e1-5ed1-44c5-a2d6-829f797c67f0-kube-api-access-2dd99\") pod \"multus-dpbg6\" (UID: \"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0\") " pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.196992 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.196358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mvj\" (UniqueName: \"kubernetes.io/projected/cfda85cb-e637-42da-a583-f28076b66cd2-kube-api-access-c9mvj\") pod \"multus-additional-cni-plugins-7cg54\" (UID: \"cfda85cb-e637-42da-a583-f28076b66cd2\") " pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.196992 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.196901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czqt\" (UniqueName: \"kubernetes.io/projected/15105daa-459e-46fd-aab4-c51a3a0ea806-kube-api-access-7czqt\") pod \"tuned-q48tf\" (UID: \"15105daa-459e-46fd-aab4-c51a3a0ea806\") " pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.197151 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.197020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2923af86-87e3-42e0-b039-7a0643df290b-ovn-node-metrics-cert\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.197685 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.197654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2nz\" (UniqueName: \"kubernetes.io/projected/e5ae6f21-9e7f-4a0e-bafb-35ad94980964-kube-api-access-hp2nz\") pod \"iptables-alerter-bttgg\" (UID: \"e5ae6f21-9e7f-4a0e-bafb-35ad94980964\") " pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.198635 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.198611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v9b4\" (UniqueName: \"kubernetes.io/projected/2923af86-87e3-42e0-b039-7a0643df290b-kube-api-access-2v9b4\") pod \"ovnkube-node-vb84k\" (UID: \"2923af86-87e3-42e0-b039-7a0643df290b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.200059 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.200038 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4f55\" (UniqueName: \"kubernetes.io/projected/08d39a33-6c3a-4dfb-a70e-9714ccc04802-kube-api-access-f4f55\") pod \"node-ca-cp4k9\" (UID: \"08d39a33-6c3a-4dfb-a70e-9714ccc04802\") " pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.238046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.238013 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"f449d6f92faa6a51a14c54944141985e","Type":"ContainerStarted","Data":"971eb9a032cc85c02beea17f450a0c1ebab19286fe5269793abeef1170d1c318"} Jul 17 21:08:49.287928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.287903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.287928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.287931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3173e73-266f-4600-a98b-97d2c3a33af7-konnectivity-ca\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.287957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.287986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjst6\" (UniqueName: \"kubernetes.io/projected/20ef74bf-c023-43cb-90f1-b8db835926b4-kube-api-access-cjst6\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.288062 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7d7\" (UniqueName: \"kubernetes.io/projected/07f27250-1057-446a-99a5-e49eb4eb1354-kube-api-access-hr7d7\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288145 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.288142 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.788120398 +0000 UTC m=+3.093119327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288176 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4js\" (UniqueName: \"kubernetes.io/projected/56fe822b-b110-43b8-bc35-b34a75fef2dd-kube-api-access-tg4js\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-kubelet-config\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-dbus\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3173e73-266f-4600-a98b-97d2c3a33af7-agent-certs\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07f27250-1057-446a-99a5-e49eb4eb1354-hosts-file\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f27250-1057-446a-99a5-e49eb4eb1354-tmp-dir\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3173e73-266f-4600-a98b-97d2c3a33af7-konnectivity-ca\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f27250-1057-446a-99a5-e49eb4eb1354-tmp-dir\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.288730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.288999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.289049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.289067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56fe822b-b110-43b8-bc35-b34a75fef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.289081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-kubelet-config\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.289161 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:49.289264 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.289207 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.789190701 +0000 UTC m=+3.094189632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:49.289547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.289426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92967c53-9967-4227-84fa-408e172b8a27-dbus\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.289547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.289533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07f27250-1057-446a-99a5-e49eb4eb1354-hosts-file\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.291815 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.291793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3173e73-266f-4600-a98b-97d2c3a33af7-agent-certs\") pod \"konnectivity-agent-r8m8k\" (UID: \"f3173e73-266f-4600-a98b-97d2c3a33af7\") " pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.295137 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.295119 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:08:49.295137 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.295140 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:08:49.295270 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.295149 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:49.295270 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.295186 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.795175927 +0000 UTC m=+3.100174836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:49.297314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.297294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7d7\" (UniqueName: \"kubernetes.io/projected/07f27250-1057-446a-99a5-e49eb4eb1354-kube-api-access-hr7d7\") pod \"node-resolver-p4ths\" (UID: \"07f27250-1057-446a-99a5-e49eb4eb1354\") " pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.297793 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.297753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4js\" (UniqueName: \"kubernetes.io/projected/56fe822b-b110-43b8-bc35-b34a75fef2dd-kube-api-access-tg4js\") pod \"aws-ebs-csi-driver-node-55t7v\" (UID: \"56fe822b-b110-43b8-bc35-b34a75fef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.297930 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.297912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjst6\" (UniqueName: \"kubernetes.io/projected/20ef74bf-c023-43cb-90f1-b8db835926b4-kube-api-access-cjst6\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.378010 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.377977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7cg54" Jul 17 21:08:49.389843 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.389824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bttgg" Jul 17 21:08:49.398788 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.398715 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:08:49.405389 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.405371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q48tf" Jul 17 21:08:49.415163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.415146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cp4k9" Jul 17 21:08:49.421743 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.421716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpbg6" Jul 17 21:08:49.429167 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.429146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" Jul 17 21:08:49.437836 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.437819 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:08:49.448419 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.448394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4ths" Jul 17 21:08:49.562338 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.562311 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:08:49.791318 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.791282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:49.791483 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.791356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:49.791483 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.791460 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:49.791483 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.791465 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:49.791630 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.791520 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:50.791505906 +0000 UTC m=+4.096504815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:49.791630 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.791534 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:50.791527923 +0000 UTC m=+4.096526832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:49.892593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:49.892557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:49.892788 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.892727 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:08:49.892788 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.892746 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:08:49.892788 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.892759 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:49.892935 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:49.892855 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:50.892834712 +0000 UTC m=+4.197833623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:50.112193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.112074 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:49 +0000 UTC" deadline="2028-04-12 00:08:54.438234586 +0000 UTC" Jul 17 21:08:50.112193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.112145 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15219h0m4.326093942s" Jul 17 21:08:50.151865 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.151841 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fe822b_b110_43b8_bc35_b34a75fef2dd.slice/crio-e053a8a17f0de36f9983d106d246fb2f881701f58e38d943081a98c3a6b3b4c7 WatchSource:0}: Error finding container e053a8a17f0de36f9983d106d246fb2f881701f58e38d943081a98c3a6b3b4c7: Status 404 returned error can't find the container with id e053a8a17f0de36f9983d106d246fb2f881701f58e38d943081a98c3a6b3b4c7 Jul 17 21:08:50.160652 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.160624 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2923af86_87e3_42e0_b039_7a0643df290b.slice/crio-9da97eda8bf568d3e0f76dc41ed4177365865f04ae3dd217bcbcbc6484d541da WatchSource:0}: Error finding container 9da97eda8bf568d3e0f76dc41ed4177365865f04ae3dd217bcbcbc6484d541da: Status 404 returned error can't find the container with id 9da97eda8bf568d3e0f76dc41ed4177365865f04ae3dd217bcbcbc6484d541da Jul 17 21:08:50.161146 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.161122 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfda85cb_e637_42da_a583_f28076b66cd2.slice/crio-4f3d8ee4a781a796710b52d5050a9d764734793f0c303f57dd4f934b7c810556 WatchSource:0}: Error finding container 4f3d8ee4a781a796710b52d5050a9d764734793f0c303f57dd4f934b7c810556: Status 404 returned error can't find the container with id 4f3d8ee4a781a796710b52d5050a9d764734793f0c303f57dd4f934b7c810556 Jul 17 21:08:50.161999 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.161976 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f27250_1057_446a_99a5_e49eb4eb1354.slice/crio-c72b88b6e8e868d8442ed0be1b98f7da97de6a81f15f98a7fafabcd2add1680d WatchSource:0}: Error finding container c72b88b6e8e868d8442ed0be1b98f7da97de6a81f15f98a7fafabcd2add1680d: Status 404 returned error can't find the container with id c72b88b6e8e868d8442ed0be1b98f7da97de6a81f15f98a7fafabcd2add1680d Jul 17 21:08:50.163341 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.163197 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeba6e1_5ed1_44c5_a2d6_829f797c67f0.slice/crio-e51c9a99e2da24168e28d54349697a60af4ec97b02f4afece7e6e0ade6563a62 WatchSource:0}: Error finding container e51c9a99e2da24168e28d54349697a60af4ec97b02f4afece7e6e0ade6563a62: Status 404 returned error can't find the container with id e51c9a99e2da24168e28d54349697a60af4ec97b02f4afece7e6e0ade6563a62 Jul 17 21:08:50.163659 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.163638 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15105daa_459e_46fd_aab4_c51a3a0ea806.slice/crio-e05f2e05648ba683907158d19049292f55639b55b97437219a0c1f4041b7a444 WatchSource:0}: Error finding container e05f2e05648ba683907158d19049292f55639b55b97437219a0c1f4041b7a444: Status 404 returned error can't find the container with id e05f2e05648ba683907158d19049292f55639b55b97437219a0c1f4041b7a444 Jul 17 21:08:50.164375 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:08:50.164354 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3173e73_266f_4600_a98b_97d2c3a33af7.slice/crio-5d9288ef5892f2149c8cc70d5116750647b2909d91ee96c8f279b0234cab3bca WatchSource:0}: Error finding container 5d9288ef5892f2149c8cc70d5116750647b2909d91ee96c8f279b0234cab3bca: Status 404 returned error can't find the container with id 5d9288ef5892f2149c8cc70d5116750647b2909d91ee96c8f279b0234cab3bca Jul 17 21:08:50.232611 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.232478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:50.232709 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.232696 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:50.241051 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.241022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cp4k9" event={"ID":"08d39a33-6c3a-4dfb-a70e-9714ccc04802","Type":"ContainerStarted","Data":"3b85bf33b3f967a2175a5d829b6025bf166eeaa6d3ba49f1c09c2279bae738f0"} Jul 17 21:08:50.242175 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.242155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bttgg" event={"ID":"e5ae6f21-9e7f-4a0e-bafb-35ad94980964","Type":"ContainerStarted","Data":"ce7fa708a707298530644539f5b54797e3b5fee1e1542cb6d965150fc7a7ebfd"} Jul 17 21:08:50.243302 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.243277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q48tf" event={"ID":"15105daa-459e-46fd-aab4-c51a3a0ea806","Type":"ContainerStarted","Data":"e05f2e05648ba683907158d19049292f55639b55b97437219a0c1f4041b7a444"} Jul 17 21:08:50.246830 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.246806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4ths" event={"ID":"07f27250-1057-446a-99a5-e49eb4eb1354","Type":"ContainerStarted","Data":"c72b88b6e8e868d8442ed0be1b98f7da97de6a81f15f98a7fafabcd2add1680d"} Jul 17 21:08:50.248896 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.248870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerStarted","Data":"4f3d8ee4a781a796710b52d5050a9d764734793f0c303f57dd4f934b7c810556"} Jul 17 21:08:50.250361 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.250340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpbg6" event={"ID":"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0","Type":"ContainerStarted","Data":"e51c9a99e2da24168e28d54349697a60af4ec97b02f4afece7e6e0ade6563a62"} Jul 17 21:08:50.251475 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.251455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" event={"ID":"56fe822b-b110-43b8-bc35-b34a75fef2dd","Type":"ContainerStarted","Data":"e053a8a17f0de36f9983d106d246fb2f881701f58e38d943081a98c3a6b3b4c7"} Jul 17 21:08:50.254412 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.254391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r8m8k" event={"ID":"f3173e73-266f-4600-a98b-97d2c3a33af7","Type":"ContainerStarted","Data":"5d9288ef5892f2149c8cc70d5116750647b2909d91ee96c8f279b0234cab3bca"} Jul 17 21:08:50.255407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.255384 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"9da97eda8bf568d3e0f76dc41ed4177365865f04ae3dd217bcbcbc6484d541da"} Jul 17 21:08:50.268452 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.268414 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" podStartSLOduration=2.26840172 podStartE2EDuration="2.26840172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:50.267942652 +0000 UTC m=+3.572941585" watchObservedRunningTime="2026-07-17 21:08:50.26840172 +0000 UTC m=+3.573400651" Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.798850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.798903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.798995 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.799063 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:52.799049029 +0000 UTC m=+6.104047937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.799312 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:50.799393 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.799401 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:52.799352106 +0000 UTC m=+6.104351017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:50.899752 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:50.899716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:50.899933 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.899904 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:08:50.899933 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.899925 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:08:50.900060 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.899937 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:50.900060 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:50.899996 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:52.899976253 +0000 UTC m=+6.204975166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:51.112924 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.112830 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:49 +0000 UTC" deadline="2028-01-11 16:02:41.266517677 +0000 UTC" Jul 17 21:08:51.112924 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.112872 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13026h53m50.153650502s" Jul 17 21:08:51.233942 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.233909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:51.234114 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:51.234044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:08:51.234584 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.234553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:51.234690 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:51.234668 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:51.265800 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.265130 2573 generic.go:356] "Generic (PLEG): container finished" podID="2cf3a63f5e16fba7955768004cdb0f7c" containerID="c353fc576f3625e0f248dc599736fb3c2a4946fec2ec96a4ebc305a52bc45e68" exitCode=0 Jul 17 21:08:51.265800 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.265213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerDied","Data":"c353fc576f3625e0f248dc599736fb3c2a4946fec2ec96a4ebc305a52bc45e68"} Jul 17 21:08:51.276848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:51.276236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"f449d6f92faa6a51a14c54944141985e","Type":"ContainerStarted","Data":"36dcf7f7a7a48e0d8c97c9f7e068b8dad29a85687607ac26ef20d1c1ba5aa78b"} Jul 17 21:08:52.232371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.232330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:52.232911 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.232470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:52.295523 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.295485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerStarted","Data":"e52863048eaf775350fa56ffc8eb31e572ffdc3116b0f6d6b12f777154029246"} Jul 17 21:08:52.317359 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.316347 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" podStartSLOduration=4.31632645 podStartE2EDuration="4.31632645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:52.316085519 +0000 UTC m=+5.621084450" watchObservedRunningTime="2026-07-17 21:08:52.31632645 +0000 UTC m=+5.621325382" Jul 17 21:08:52.818301 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.818259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:52.818482 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.818344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:52.818586 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.818489 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:52.818586 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.818551 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:56.818531991 +0000 UTC m=+10.123530904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:52.819000 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.818979 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:52.819075 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.819042 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:56.819026116 +0000 UTC m=+10.124025026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:52.919397 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:52.919356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:52.919558 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.919523 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:08:52.919558 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.919541 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:08:52.919558 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.919553 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:52.919716 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:52.919610 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:56.919592234 +0000 UTC m=+10.224591146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:53.234972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:53.234892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:53.235398 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:53.235022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:08:53.235398 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:53.235381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:53.235507 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:53.235469 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:54.232590 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:54.232555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:54.232745 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:54.232684 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:55.233925 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:55.233889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:55.234358 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:55.233887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:55.234358 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:55.234022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:55.234358 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:55.234116 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:08:56.232549 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:56.232511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:56.232818 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.232784 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:56.848997 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:56.848949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:56.849426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:56.849041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:56.849426 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.849182 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:56.849426 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.849246 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:04.849227688 +0000 UTC m=+18.154226604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:08:56.849830 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.849617 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:56.849830 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.849657 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:04.849647106 +0000 UTC m=+18.154646015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:08:56.949594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:56.949531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:56.949752 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.949734 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:08:56.949830 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.949759 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:08:56.949830 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.949788 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:56.949920 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:56.949851 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:04.949831015 +0000 UTC m=+18.254829927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:08:57.233793 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:57.233696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:57.233948 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:57.233845 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:57.234285 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:57.234264 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:57.234366 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:57.234347 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:08:58.233824 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:58.232711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:08:58.233824 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:58.232873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:08:59.232915 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:59.232405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:08:59.232915 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:08:59.232407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:08:59.232915 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:59.232543 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:08:59.232915 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:08:59.232596 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:00.233254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:00.232828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:00.233254 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:00.232949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:01.232689 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:01.232604 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:01.232859 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:01.232704 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:01.232859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:01.232784 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:01.232967 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:01.232899 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:02.232493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:02.232451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:02.232948 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:02.232571 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:03.233218 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:03.233184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:03.233673 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:03.233306 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:03.233673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:03.233366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:03.233673 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:03.233469 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:04.232659 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:04.232616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:04.232899 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:04.232743 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:04.913008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:04.912918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:04.913008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:04.912992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:04.913491 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:04.913083 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:09:04.913491 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:04.913091 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:09:04.913491 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:04.913165 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:20.913142865 +0000 UTC m=+34.218141790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:09:04.913491 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:04.913186 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:20.913175559 +0000 UTC m=+34.218174474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:09:05.013972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:05.013929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:05.014153 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.014070 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:09:05.014153 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.014088 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:09:05.014153 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.014103 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:09:05.014332 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.014164 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:21.014143683 +0000 UTC m=+34.319142607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:09:05.232990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:05.232913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:05.233161 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.233087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:05.233219 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:05.233160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:05.233316 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:05.233272 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:06.232945 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:06.232910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:06.233318 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:06.233056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:07.233133 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:07.233091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:07.233584 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:07.233191 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:07.233584 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:07.233446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:07.233584 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:07.233567 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:08.232764 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:08.232723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:08.232948 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:08.232866 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:09.232492 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.232339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:09.233152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.232408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:09.233152 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:09.232557 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:09.233152 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:09.232658 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:09.336275 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.336251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q48tf" event={"ID":"15105daa-459e-46fd-aab4-c51a3a0ea806","Type":"ContainerStarted","Data":"e8ec4aefd5d5a44110ea5b9864d8dce7f0f15e2ceeefe537f3054feadbdfaa3b"} Jul 17 21:09:09.337940 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.337907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4ths" event={"ID":"07f27250-1057-446a-99a5-e49eb4eb1354","Type":"ContainerStarted","Data":"14ef950a600bf5aa50b174ce44699d33927b8c642757d1ea0e1a79a027dfb309"} Jul 17 21:09:09.339441 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.339418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerStarted","Data":"8d8e20444cc05ab52b1d24dd0187574849f99e5e343caccd03ae42c9d1eacda6"} Jul 17 21:09:09.341581 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.341557 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpbg6" event={"ID":"bbeba6e1-5ed1-44c5-a2d6-829f797c67f0","Type":"ContainerStarted","Data":"cc1570615d97ac7a384dc56c9deab8c9a87c0cc7579c3bc263e6fa7713d689ba"} Jul 17 21:09:09.343339 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.343310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" event={"ID":"56fe822b-b110-43b8-bc35-b34a75fef2dd","Type":"ContainerStarted","Data":"e5da2d65d048adc4d358fef0241703379a90ef5d70aa55890b0e4906635db609"} Jul 17 21:09:09.344954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.344926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r8m8k" event={"ID":"f3173e73-266f-4600-a98b-97d2c3a33af7","Type":"ContainerStarted","Data":"108522f8d6a16243de79cf8befebf0f2d2d056d4e8016a5fbe31c7eabb713b06"} Jul 17 21:09:09.347471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.347454 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:09:09.347902 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.347872 2573 generic.go:356] "Generic (PLEG): container finished" podID="2923af86-87e3-42e0-b039-7a0643df290b" containerID="802089e4ae46931a77195e8de3cd0045c1297254a553defcf74554e7c84fcb66" exitCode=1 Jul 17 21:09:09.348003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.347956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerDied","Data":"802089e4ae46931a77195e8de3cd0045c1297254a553defcf74554e7c84fcb66"} Jul 17 21:09:09.348067 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.348014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"f3f55c03c94c2201aef1dd3af6934a59e077b47d29b21d222a5271e4bfa9e01e"} Jul 17 21:09:09.349856 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.349833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cp4k9" event={"ID":"08d39a33-6c3a-4dfb-a70e-9714ccc04802","Type":"ContainerStarted","Data":"84e454579cf1c95c1287780e3c1e8fa33a62d917b02ab0ae9ab31e0281a54696"} Jul 17 21:09:09.356123 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.356080 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q48tf" podStartSLOduration=3.579616422 podStartE2EDuration="22.356068159s" totalImagesPullingTime="18.776451737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:09.355674546 +0000 UTC m=+22.660673506" watchObservedRunningTime="2026-07-17 21:09:09.356068159 +0000 UTC m=+22.661067091" Jul 17 21:09:09.413692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.413642 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cp4k9" podStartSLOduration=3.642527532 podStartE2EDuration="22.413626754s" totalImagesPullingTime="18.771099222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:09.395351759 +0000 UTC m=+22.700350689" watchObservedRunningTime="2026-07-17 21:09:09.413626754 +0000 UTC m=+22.718625708" Jul 17 21:09:09.414085 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.414052 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p4ths" podStartSLOduration=3.637402361 podStartE2EDuration="22.414040798s" totalImagesPullingTime="18.776638437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:09.413877426 +0000 UTC m=+22.718876349" watchObservedRunningTime="2026-07-17 21:09:09.414040798 +0000 UTC m=+22.719039729" Jul 17 21:09:09.433367 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.433316 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dpbg6" podStartSLOduration=3.606395317 podStartE2EDuration="22.433297067s" totalImagesPullingTime="18.82690175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:09.433268672 +0000 UTC m=+22.738267603" watchObservedRunningTime="2026-07-17 21:09:09.433297067 +0000 UTC m=+22.738296192" Jul 17 21:09:09.448864 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:09.448790 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r8m8k" podStartSLOduration=8.032646408 podStartE2EDuration="22.448755372s" totalImagesPullingTime="14.416108964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:09.448567839 +0000 UTC m=+22.753566770" watchObservedRunningTime="2026-07-17 21:09:09.448755372 +0000 UTC m=+22.753754293" Jul 17 21:09:10.232865 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.232643 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:10.233219 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:10.232901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:10.354662 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.354631 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:09:10.355116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.355083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"1d5a60482d08cffd790b717f55d445360ca748437911d52b527a6991627f60fe"} Jul 17 21:09:10.355246 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.355178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"615c58587d15cd84117e5a39c70bc4c773dc7d5884d87108c08706827779c094"} Jul 17 21:09:10.355246 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.355232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"1bad48081d6d727e0951cea1c333d819129ac0e571f68cd434ff07671d54c159"} Jul 17 21:09:10.355346 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.355278 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"9f511d21dd4adf35b92c6ebb9cbba0f1c00148b9dd5cfa945afb111313c0fac0"} Jul 17 21:09:10.356509 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.356478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bttgg" event={"ID":"e5ae6f21-9e7f-4a0e-bafb-35ad94980964","Type":"ContainerStarted","Data":"45a7fa8e85dafc948823e0d0b6aa0bd9e4ac06321daa9b68634c90c217c87942"} Jul 17 21:09:10.357803 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.357764 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="8d8e20444cc05ab52b1d24dd0187574849f99e5e343caccd03ae42c9d1eacda6" exitCode=0 Jul 17 21:09:10.357803 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.357795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"8d8e20444cc05ab52b1d24dd0187574849f99e5e343caccd03ae42c9d1eacda6"} Jul 17 21:09:10.372221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.372182 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bttgg" podStartSLOduration=8.959161543 podStartE2EDuration="23.372166039s" totalImagesPullingTime="14.413004496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:10.371716197 +0000 UTC m=+23.676715128" watchObservedRunningTime="2026-07-17 21:09:10.372166039 +0000 UTC m=+23.677164969" Jul 17 21:09:10.610146 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:10.610116 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 21:09:11.116371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.116332 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:09:11.150788 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.150589 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:09:10.610141572Z","UUID":"223075a2-7826-4b7c-aaf8-898606f8fbe3","Handler":null,"Name":"","Endpoint":""} Jul 17 21:09:11.152722 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.152699 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 21:09:11.152846 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.152742 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 21:09:11.171228 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.171205 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:09:11.172171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.172150 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:09:11.232414 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.232391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:11.232549 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.232396 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:11.232549 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:11.232498 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:11.232665 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:11.232593 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:11.362129 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.362089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" event={"ID":"56fe822b-b110-43b8-bc35-b34a75fef2dd","Type":"ContainerStarted","Data":"35213adef7ff4c67a81cde1c49b4cee79aaf01627a9a1e861ea644d1c357960a"} Jul 17 21:09:11.362955 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:11.362935 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r8m8k" Jul 17 21:09:12.232674 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:12.232447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:12.232858 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:12.232813 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:12.367324 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:12.367245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" event={"ID":"56fe822b-b110-43b8-bc35-b34a75fef2dd","Type":"ContainerStarted","Data":"d57219d50c2783842a823a4d3be7008e222d4c309e36cce9c25c8cea79c04240"} Jul 17 21:09:12.374562 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:12.374536 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:09:12.375008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:12.374986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"d11326d33afb91101176f795ff9764451eec5e7017c6f3c2031a4134db52d87b"} Jul 17 21:09:12.387136 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:12.387083 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55t7v" podStartSLOduration=4.025645993 podStartE2EDuration="25.387062459s" totalImagesPullingTime="21.361416466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:12.386093227 +0000 UTC m=+25.691092157" watchObservedRunningTime="2026-07-17 21:09:12.387062459 +0000 UTC m=+25.692061391" Jul 17 21:09:13.232515 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:13.232486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:13.232738 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:13.232529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:13.232738 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:13.232613 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:13.232738 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:13.232704 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:14.232821 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:14.232788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:14.233322 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:14.232903 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:15.233107 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.233027 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:15.233668 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:15.233158 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:15.233668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.233242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:15.233668 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:15.233334 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:15.384746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.384444 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:09:15.385208 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.384803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"45b49cea3aa013bffdf984a50d5c0176514e4b92cba4540d13d193bcfc86dea5"} Jul 17 21:09:15.385208 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.385087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:15.385795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.385214 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:15.385795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.385268 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:15.385795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.385361 2573 scope.go:117] "RemoveContainer" containerID="802089e4ae46931a77195e8de3cd0045c1297254a553defcf74554e7c84fcb66" Jul 17 21:09:15.403970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.403620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:15.404380 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:15.404255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:16.232530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.232498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:16.232923 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:16.232624 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:16.388285 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.388253 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="61ba73899ec81fa0378e31304b68e497a9c338bfc76dce1f626d27b04757b64b" exitCode=0 Jul 17 21:09:16.388729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.388345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"61ba73899ec81fa0378e31304b68e497a9c338bfc76dce1f626d27b04757b64b"} Jul 17 21:09:16.392079 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.392064 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:09:16.392412 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.392389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" event={"ID":"2923af86-87e3-42e0-b039-7a0643df290b","Type":"ContainerStarted","Data":"7f3bb9fcad61b222df940cf3c32d82a3f3959400f8c5972ee6d722bfdd816d48"} Jul 17 21:09:16.436219 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:16.436185 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" podStartSLOduration=10.5926994 podStartE2EDuration="29.43617306s" totalImagesPullingTime="18.84347366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:16.435785528 +0000 UTC m=+29.740784462" watchObservedRunningTime="2026-07-17 21:09:16.43617306 +0000 UTC m=+29.741171990" Jul 17 21:09:17.233453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:17.233425 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:17.233602 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:17.233543 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:17.233602 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:17.233589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:17.233688 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:17.233668 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:18.233221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.233088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:18.233498 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:18.233303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:18.399283 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.399247 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="b771b07e9e7c71ba3d248fcdee3fe05bfa8667f7fc548a962c2c81c7f9f73b5a" exitCode=0 Jul 17 21:09:18.399398 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.399301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"b771b07e9e7c71ba3d248fcdee3fe05bfa8667f7fc548a962c2c81c7f9f73b5a"} Jul 17 21:09:18.437939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.437906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k2hd6"] Jul 17 21:09:18.438082 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.438007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:18.438185 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:18.438156 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:18.442358 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.442333 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hnhvt"] Jul 17 21:09:18.442507 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.442441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:18.442560 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:18.442540 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:18.442993 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.442922 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w929x"] Jul 17 21:09:18.443100 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:18.443001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:18.443100 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:18.443080 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:20.232831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.232803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:20.233449 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.232797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:20.233449 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.232906 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w929x" podUID="167e7e67-7a9e-47ad-a05a-7478eedc6602" Jul 17 21:09:20.233449 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.232917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:20.233449 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.232983 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhvt" podUID="20ef74bf-c023-43cb-90f1-b8db835926b4" Jul 17 21:09:20.233449 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.233047 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2hd6" podUID="92967c53-9967-4227-84fa-408e172b8a27" Jul 17 21:09:20.405985 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.405952 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="3682aab3a9a1e9562d0812487878dc900f6ef67497ee516e2b7062e90cab8b28" exitCode=0 Jul 17 21:09:20.406122 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.406006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"3682aab3a9a1e9562d0812487878dc900f6ef67497ee516e2b7062e90cab8b28"} Jul 17 21:09:20.937576 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.937541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:20.937763 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:20.937627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:20.937763 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.937703 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:09:20.937763 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.937733 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:09:20.937946 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.937793 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs podName:20ef74bf-c023-43cb-90f1-b8db835926b4 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:52.937754282 +0000 UTC m=+66.242753191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs") pod "network-metrics-daemon-hnhvt" (UID: "20ef74bf-c023-43cb-90f1-b8db835926b4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:09:20.937946 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:20.937812 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret podName:92967c53-9967-4227-84fa-408e172b8a27 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:52.937802889 +0000 UTC m=+66.242801798 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret") pod "global-pull-secret-syncer-k2hd6" (UID: "92967c53-9967-4227-84fa-408e172b8a27") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:09:21.038599 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:21.038568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:21.038803 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:21.038723 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:09:21.038803 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:21.038742 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:09:21.038803 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:21.038756 2573 projected.go:196] Error preparing data for projected volume kube-api-access-9czzz for pod openshift-network-diagnostics/network-check-target-w929x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:09:21.038928 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:21.038831 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz podName:167e7e67-7a9e-47ad-a05a-7478eedc6602 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:53.038816645 +0000 UTC m=+66.343815554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9czzz" (UniqueName: "kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz") pod "network-check-target-w929x" (UID: "167e7e67-7a9e-47ad-a05a-7478eedc6602") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:09:22.055307 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.055273 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeReady" Jul 17 21:09:22.055809 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.055408 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 21:09:22.103069 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.103039 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8nvrx"] Jul 17 21:09:22.106276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.105112 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.106276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.105539 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z6l26"] Jul 17 21:09:22.107256 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.107236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.107675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.107652 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\"" Jul 17 21:09:22.107763 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.107715 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 21:09:22.108572 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.108554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 21:09:22.109747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.109729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\"" Jul 17 21:09:22.109862 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.109746 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 21:09:22.110055 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.110036 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 21:09:22.110142 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.110111 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 21:09:22.110202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.110111 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 21:09:22.121269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.121248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8nvrx"] Jul 17 21:09:22.131913 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.131892 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gg29x"] Jul 17 21:09:22.133807 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.133788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.136917 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.136898 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 21:09:22.137148 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.136919 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\"" Jul 17 21:09:22.137338 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.137315 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 21:09:22.137435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.137405 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 21:09:22.137985 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.137964 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z6l26"] Jul 17 21:09:22.146476 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.146455 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gg29x"] Jul 17 21:09:22.232223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.232188 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:22.232377 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.232241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:22.232377 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.232201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:22.235035 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235016 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:09:22.235147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235129 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\"" Jul 17 21:09:22.235224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235191 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:09:22.235224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235204 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:09:22.235380 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235363 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:09:22.235419 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.235396 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\"" Jul 17 21:09:22.247048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-tmp-dir\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.247157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99090a94-5102-43d5-911d-dd9aa6f8a288-data-volume\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.247157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gmpd\" (UniqueName: \"kubernetes.io/projected/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-api-access-4gmpd\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.247157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99090a94-5102-43d5-911d-dd9aa6f8a288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.247157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzpm\" (UniqueName: \"kubernetes.io/projected/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-kube-api-access-xfzpm\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99090a94-5102-43d5-911d-dd9aa6f8a288-crio-socket\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpmr\" (UniqueName: \"kubernetes.io/projected/d7fd4996-7d69-4e26-a89b-ca075b4b457e-kube-api-access-nwpmr\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-metrics-tls\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-config-volume\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.247407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.247271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7fd4996-7d69-4e26-a89b-ca075b4b457e-cert\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.348212 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gmpd\" (UniqueName: \"kubernetes.io/projected/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-api-access-4gmpd\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348212 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99090a94-5102-43d5-911d-dd9aa6f8a288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348212 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzpm\" (UniqueName: \"kubernetes.io/projected/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-kube-api-access-xfzpm\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99090a94-5102-43d5-911d-dd9aa6f8a288-crio-socket\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpmr\" (UniqueName: \"kubernetes.io/projected/d7fd4996-7d69-4e26-a89b-ca075b4b457e-kube-api-access-nwpmr\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-metrics-tls\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-config-volume\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7fd4996-7d69-4e26-a89b-ca075b4b457e-cert\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-tmp-dir\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.348459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99090a94-5102-43d5-911d-dd9aa6f8a288-data-volume\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348802 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99090a94-5102-43d5-911d-dd9aa6f8a288-crio-socket\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.348802 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99090a94-5102-43d5-911d-dd9aa6f8a288-data-volume\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.349029 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.348989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-tmp-dir\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.349029 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.349008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-config-volume\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.349638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.349395 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.352646 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.352623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-metrics-tls\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.352748 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.352654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99090a94-5102-43d5-911d-dd9aa6f8a288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.352918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.352900 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7fd4996-7d69-4e26-a89b-ca075b4b457e-cert\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.357154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.357135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gmpd\" (UniqueName: \"kubernetes.io/projected/99090a94-5102-43d5-911d-dd9aa6f8a288-kube-api-access-4gmpd\") pod \"insights-runtime-extractor-z6l26\" (UID: \"99090a94-5102-43d5-911d-dd9aa6f8a288\") " pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.357724 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.357706 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpmr\" (UniqueName: \"kubernetes.io/projected/d7fd4996-7d69-4e26-a89b-ca075b4b457e-kube-api-access-nwpmr\") pod \"ingress-canary-gg29x\" (UID: \"d7fd4996-7d69-4e26-a89b-ca075b4b457e\") " pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.358215 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.358198 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzpm\" (UniqueName: \"kubernetes.io/projected/67f6c1d5-9932-4936-a7a3-18ccb9364ddd-kube-api-access-xfzpm\") pod \"dns-default-8nvrx\" (UID: \"67f6c1d5-9932-4936-a7a3-18ccb9364ddd\") " pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.418094 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.418062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:22.425285 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.425261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z6l26" Jul 17 21:09:22.443267 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.443235 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gg29x" Jul 17 21:09:22.598983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.598645 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gg29x"] Jul 17 21:09:22.601465 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.601389 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8nvrx"] Jul 17 21:09:22.604686 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:22.604658 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f6c1d5_9932_4936_a7a3_18ccb9364ddd.slice/crio-c93cf382726b2a3865bd0e2db332697ab7d0c35e3f39434459af3b4c19f719c8 WatchSource:0}: Error finding container c93cf382726b2a3865bd0e2db332697ab7d0c35e3f39434459af3b4c19f719c8: Status 404 returned error can't find the container with id c93cf382726b2a3865bd0e2db332697ab7d0c35e3f39434459af3b4c19f719c8 Jul 17 21:09:22.605107 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:22.605086 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fd4996_7d69_4e26_a89b_ca075b4b457e.slice/crio-aed4ba1592f9be1c8768340296be00f74adc506f59fe64b007b6e1582d1feb41 WatchSource:0}: Error finding container aed4ba1592f9be1c8768340296be00f74adc506f59fe64b007b6e1582d1feb41: Status 404 returned error can't find the container with id aed4ba1592f9be1c8768340296be00f74adc506f59fe64b007b6e1582d1feb41 Jul 17 21:09:22.611581 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:22.611558 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z6l26"] Jul 17 21:09:22.614839 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:22.614820 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99090a94_5102_43d5_911d_dd9aa6f8a288.slice/crio-7d30f736a369029334f34a518cf64ee7d07707d11282d333b9be6f8643e84883 WatchSource:0}: Error finding container 7d30f736a369029334f34a518cf64ee7d07707d11282d333b9be6f8643e84883: Status 404 returned error can't find the container with id 7d30f736a369029334f34a518cf64ee7d07707d11282d333b9be6f8643e84883 Jul 17 21:09:23.416841 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:23.416728 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gg29x" event={"ID":"d7fd4996-7d69-4e26-a89b-ca075b4b457e","Type":"ContainerStarted","Data":"aed4ba1592f9be1c8768340296be00f74adc506f59fe64b007b6e1582d1feb41"} Jul 17 21:09:23.419900 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:23.419786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z6l26" event={"ID":"99090a94-5102-43d5-911d-dd9aa6f8a288","Type":"ContainerStarted","Data":"7d30f736a369029334f34a518cf64ee7d07707d11282d333b9be6f8643e84883"} Jul 17 21:09:23.422910 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:23.422883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nvrx" event={"ID":"67f6c1d5-9932-4936-a7a3-18ccb9364ddd","Type":"ContainerStarted","Data":"c93cf382726b2a3865bd0e2db332697ab7d0c35e3f39434459af3b4c19f719c8"} Jul 17 21:09:26.170938 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.170905 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:09:26.189312 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.189290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:09:26.189453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.189382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.192288 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.192261 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 21:09:26.192431 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.192356 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 21:09:26.192431 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.192395 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 21:09:26.193404 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.193383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-526lw\"" Jul 17 21:09:26.193504 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.193459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 21:09:26.193561 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.193383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 21:09:26.194236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.193683 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 21:09:26.194236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.193826 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 21:09:26.381828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381785 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.381828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.382037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.382037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqbn\" (UniqueName: \"kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.382037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.382037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.381987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482568 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482568 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482568 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482568 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqbn\" (UniqueName: \"kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.482891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.482654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.483231 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.483196 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.483319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.483293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.483319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.483302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.485819 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.485801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.485969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.485950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.491045 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.491020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqbn\" (UniqueName: \"kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn\") pod \"console-7b6b684c8-2wkpw\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:26.500708 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:26.500686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:31.077511 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.077485 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:09:31.082701 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:31.082674 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84afa984_e4ab_43dd_ba34_26ae129da158.slice/crio-f0bacdee39f8684f87919c631fa8837f36becf3ba055a66c8ffce939c80fa98d WatchSource:0}: Error finding container f0bacdee39f8684f87919c631fa8837f36becf3ba055a66c8ffce939c80fa98d: Status 404 returned error can't find the container with id f0bacdee39f8684f87919c631fa8837f36becf3ba055a66c8ffce939c80fa98d Jul 17 21:09:31.444830 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.444799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6b684c8-2wkpw" event={"ID":"84afa984-e4ab-43dd-ba34-26ae129da158","Type":"ContainerStarted","Data":"f0bacdee39f8684f87919c631fa8837f36becf3ba055a66c8ffce939c80fa98d"} Jul 17 21:09:31.447765 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.447738 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="40598c35833232b602fcf3fcd49df5ef14db671918b758f5037683adddec06a0" exitCode=0 Jul 17 21:09:31.447887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.447817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"40598c35833232b602fcf3fcd49df5ef14db671918b758f5037683adddec06a0"} Jul 17 21:09:31.449316 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.449297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gg29x" event={"ID":"d7fd4996-7d69-4e26-a89b-ca075b4b457e","Type":"ContainerStarted","Data":"1067a223843157c7bb9eb626f82fbf8d0bb7e4bf4e6c167373849423ce3e7521"} Jul 17 21:09:31.450864 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.450844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z6l26" event={"ID":"99090a94-5102-43d5-911d-dd9aa6f8a288","Type":"ContainerStarted","Data":"d0460b58fd016b83dc14c2757466f8949f25888b90fda2b8728023f76dbb4bd6"} Jul 17 21:09:31.452523 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.452504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nvrx" event={"ID":"67f6c1d5-9932-4936-a7a3-18ccb9364ddd","Type":"ContainerStarted","Data":"6fe01f1666e1443c94b81c0265c31d828f37fe479ce108f31a1476053248db3b"} Jul 17 21:09:31.452636 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.452541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nvrx" event={"ID":"67f6c1d5-9932-4936-a7a3-18ccb9364ddd","Type":"ContainerStarted","Data":"f5533e23969eee5069b3cee7cd8e331408bdf5de2d75d08a6fbfbd2f2d98cb0a"} Jul 17 21:09:31.452692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.452639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:31.504056 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.504009 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8nvrx" podStartSLOduration=1.171867945 podStartE2EDuration="9.503994063s" totalImagesPullingTime="8.332126118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:31.487919976 +0000 UTC m=+44.792918906" watchObservedRunningTime="2026-07-17 21:09:31.503994063 +0000 UTC m=+44.808993005" Jul 17 21:09:31.504300 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:31.504258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gg29x" podStartSLOduration=1.1722527440000001 podStartE2EDuration="9.504250871s" totalImagesPullingTime="8.331998127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:31.503150672 +0000 UTC m=+44.808149604" watchObservedRunningTime="2026-07-17 21:09:31.504250871 +0000 UTC m=+44.809249805" Jul 17 21:09:32.460389 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:32.460356 2573 generic.go:356] "Generic (PLEG): container finished" podID="cfda85cb-e637-42da-a583-f28076b66cd2" containerID="8fe7f199e90522dfaafeeeefef4e3421d65766d4427e32c275a9cd43f2201ed3" exitCode=0 Jul 17 21:09:32.460868 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:32.460424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerDied","Data":"8fe7f199e90522dfaafeeeefef4e3421d65766d4427e32c275a9cd43f2201ed3"} Jul 17 21:09:33.468937 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:33.468896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cg54" event={"ID":"cfda85cb-e637-42da-a583-f28076b66cd2","Type":"ContainerStarted","Data":"9644a9fb0a0e7dac3badc27ab0280f0e5815bbd4d5ebba4214677a25448c8359"} Jul 17 21:09:33.493334 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:33.493285 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7cg54" podStartSLOduration=10.086712274 podStartE2EDuration="46.4932659s" totalImagesPullingTime="36.406553626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:33.491764496 +0000 UTC m=+46.796763429" watchObservedRunningTime="2026-07-17 21:09:33.4932659 +0000 UTC m=+46.798264834" Jul 17 21:09:34.758820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.758786 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:09:34.798434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.798405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:09:34.798696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.798498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.806359 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.806325 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 21:09:34.842851 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.842816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.842864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2q2\" (UniqueName: \"kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.842952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.842990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.843018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.843094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.843140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.843121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944062 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944235 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944301 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944301 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2q2\" (UniqueName: \"kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944405 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944405 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944405 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.944849 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.944759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.945349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.945323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.945472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.945420 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.945536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.945517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.947263 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.947217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.947377 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.947314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:34.953042 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:34.953013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2q2\" (UniqueName: \"kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2\") pod \"console-589dd559dc-t59lk\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:35.107115 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:35.107038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:37.049463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.049430 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-225wh"] Jul 17 21:09:37.108351 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.108315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-225wh"] Jul 17 21:09:37.108495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.108443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.111114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.111089 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 21:09:37.112274 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.112253 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 21:09:37.112411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.112301 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 21:09:37.112411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.112307 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 21:09:37.112411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.112376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4ml75\"" Jul 17 21:09:37.112641 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.112626 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 21:09:37.160848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.160822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41790925-1420-4927-b9a8-dd8248642037-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.160970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.160849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w8d\" (UniqueName: \"kubernetes.io/projected/41790925-1420-4927-b9a8-dd8248642037-kube-api-access-x4w8d\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.160970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.160880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.161044 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.160975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.262033 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.262000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41790925-1420-4927-b9a8-dd8248642037-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.262033 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.262035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w8d\" (UniqueName: \"kubernetes.io/projected/41790925-1420-4927-b9a8-dd8248642037-kube-api-access-x4w8d\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.262236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.262069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.262236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.262117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.262712 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.262684 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41790925-1420-4927-b9a8-dd8248642037-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.264612 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.264588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.264707 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.264663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41790925-1420-4927-b9a8-dd8248642037-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.270791 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.270757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w8d\" (UniqueName: \"kubernetes.io/projected/41790925-1420-4927-b9a8-dd8248642037-kube-api-access-x4w8d\") pod \"prometheus-operator-5945b5cfd-225wh\" (UID: \"41790925-1420-4927-b9a8-dd8248642037\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.418171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.418147 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" Jul 17 21:09:37.428739 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.428713 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:09:37.430986 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:37.430957 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa78987_1413_4a14_9fde_1e1f335df65b.slice/crio-e908fca1c299cc50a92ab69ec2ac2b7064a55232c8e8ab568ac7bf1d70158035 WatchSource:0}: Error finding container e908fca1c299cc50a92ab69ec2ac2b7064a55232c8e8ab568ac7bf1d70158035: Status 404 returned error can't find the container with id e908fca1c299cc50a92ab69ec2ac2b7064a55232c8e8ab568ac7bf1d70158035 Jul 17 21:09:37.490578 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.490551 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6b684c8-2wkpw" event={"ID":"84afa984-e4ab-43dd-ba34-26ae129da158","Type":"ContainerStarted","Data":"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94"} Jul 17 21:09:37.493061 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.493002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589dd559dc-t59lk" event={"ID":"2fa78987-1413-4a14-9fde-1e1f335df65b","Type":"ContainerStarted","Data":"e908fca1c299cc50a92ab69ec2ac2b7064a55232c8e8ab568ac7bf1d70158035"} Jul 17 21:09:37.513101 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.513052 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b6b684c8-2wkpw" podStartSLOduration=5.289342342 podStartE2EDuration="11.513036504s" totalImagesPullingTime="6.223694162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:37.512472975 +0000 UTC m=+50.817471906" watchObservedRunningTime="2026-07-17 21:09:37.513036504 +0000 UTC m=+50.818035435" Jul 17 21:09:37.556721 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:37.556692 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-225wh"] Jul 17 21:09:37.622000 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:37.621920 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41790925_1420_4927_b9a8_dd8248642037.slice/crio-394ac469f16bbd77ebcccb39530e0310711f8c7806e518d98d330b718487e87e WatchSource:0}: Error finding container 394ac469f16bbd77ebcccb39530e0310711f8c7806e518d98d330b718487e87e: Status 404 returned error can't find the container with id 394ac469f16bbd77ebcccb39530e0310711f8c7806e518d98d330b718487e87e Jul 17 21:09:38.499374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:38.499334 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z6l26" event={"ID":"99090a94-5102-43d5-911d-dd9aa6f8a288","Type":"ContainerStarted","Data":"dc12edc5aee5561f52f44572b2765b92312c6acb4cf8c43734f9ab74f58e3190"} Jul 17 21:09:38.502198 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:38.501832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589dd559dc-t59lk" event={"ID":"2fa78987-1413-4a14-9fde-1e1f335df65b","Type":"ContainerStarted","Data":"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175"} Jul 17 21:09:38.503425 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:38.503368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" event={"ID":"41790925-1420-4927-b9a8-dd8248642037","Type":"ContainerStarted","Data":"394ac469f16bbd77ebcccb39530e0310711f8c7806e518d98d330b718487e87e"} Jul 17 21:09:38.519579 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:38.519532 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-589dd559dc-t59lk" podStartSLOduration=4.519513902 podStartE2EDuration="4.519513902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:38.518202342 +0000 UTC m=+51.823201276" watchObservedRunningTime="2026-07-17 21:09:38.519513902 +0000 UTC m=+51.824512834" Jul 17 21:09:40.513044 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:40.513005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" event={"ID":"41790925-1420-4927-b9a8-dd8248642037","Type":"ContainerStarted","Data":"ea54d34c1e462cd61004b0d78331e441c17b65faa3a23cb39baca6e17ea96f87"} Jul 17 21:09:40.513481 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:40.513061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" event={"ID":"41790925-1420-4927-b9a8-dd8248642037","Type":"ContainerStarted","Data":"c43a29112bc9475d58c32c7fc895826dbdd902a23fd710dcf3da69208464dde9"} Jul 17 21:09:40.528976 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:40.528926 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-225wh" podStartSLOduration=1.636804192 podStartE2EDuration="3.528887711s" totalImagesPullingTime="1.892083519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:40.527909976 +0000 UTC m=+53.832908908" watchObservedRunningTime="2026-07-17 21:09:40.528887711 +0000 UTC m=+53.833886641" Jul 17 21:09:41.463018 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:41.462978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8nvrx" Jul 17 21:09:42.420060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.420030 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr"] Jul 17 21:09:42.422010 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.421995 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.424594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.424577 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:09:42.424683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.424577 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 21:09:42.425193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.425179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zxvx6\"" Jul 17 21:09:42.440101 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.440079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr"] Jul 17 21:09:42.478239 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.478210 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvd5z"] Jul 17 21:09:42.480471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.480447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.483371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.483353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 21:09:42.483482 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.483433 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:09:42.483878 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.483857 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2nzg7\"" Jul 17 21:09:42.483985 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.483963 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 21:09:42.496349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.496332 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvd5z"] Jul 17 21:09:42.497044 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.497029 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6qhm8"] Jul 17 21:09:42.499094 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.499075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.503890 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.503874 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 21:09:42.504453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.504287 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 21:09:42.504453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.504316 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 21:09:42.504579 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.504508 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\"" Jul 17 21:09:42.506423 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.506523 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.506573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/55f622da-ecd7-4413-8629-8cce3056b0ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.506611 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.506643 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.506687 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2j2s\" (UniqueName: \"kubernetes.io/projected/8eb1e02e-a084-43d6-be49-889cc71b330c-kube-api-access-r2j2s\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.506728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.506728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e02e-a084-43d6-be49-889cc71b330c-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.506829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.506829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.506813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzfh\" (UniqueName: \"kubernetes.io/projected/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-api-access-xtzfh\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.524428 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.524406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z6l26" event={"ID":"99090a94-5102-43d5-911d-dd9aa6f8a288","Type":"ContainerStarted","Data":"3cdf6863d051ccc6620191b999acc701252fb8e150f86766f731f500594f9c60"} Jul 17 21:09:42.562488 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.562446 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z6l26" podStartSLOduration=1.594142245 podStartE2EDuration="20.562430027s" totalImagesPullingTime="18.968287782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:09:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:42.562080961 +0000 UTC m=+55.867079911" watchObservedRunningTime="2026-07-17 21:09:42.562430027 +0000 UTC m=+55.867428960" Jul 17 21:09:42.607428 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-wtmp\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.607534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.607534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzfh\" (UniqueName: \"kubernetes.io/projected/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-api-access-xtzfh\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.607534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/9111c994-45fe-4cf1-ac20-3f2562da51c6-kube-api-access-gg8fl\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:42.607607 2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-metrics-client-ca\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:42.607668 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls podName:55f622da-ecd7-4413-8629-8cce3056b0ad nodeName:}" failed. No retries permitted until 2026-07-17 21:09:43.107647803 +0000 UTC m=+56.412646726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-cvd5z" (UID: "55f622da-ecd7-4413-8629-8cce3056b0ad") : secret "kube-state-metrics-tls" not found Jul 17 21:09:42.607696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-root\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/55f622da-ecd7-4413-8629-8cce3056b0ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-textfile\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.607963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.608068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2j2s\" (UniqueName: \"kubernetes.io/projected/8eb1e02e-a084-43d6-be49-889cc71b330c-kube-api-access-r2j2s\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.608434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.608434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/55f622da-ecd7-4413-8629-8cce3056b0ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.608434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-sys\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.608434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e02e-a084-43d6-be49-889cc71b330c-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.608434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.608644 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.609052 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.608995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e02e-a084-43d6-be49-889cc71b330c-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.610574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.610546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.610713 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.610695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8eb1e02e-a084-43d6-be49-889cc71b330c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.610816 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.610794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.618280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.618259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzfh\" (UniqueName: \"kubernetes.io/projected/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-api-access-xtzfh\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:42.618736 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.618720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2j2s\" (UniqueName: \"kubernetes.io/projected/8eb1e02e-a084-43d6-be49-889cc71b330c-kube-api-access-r2j2s\") pod \"openshift-state-metrics-79f6958cdf-2fvbr\" (UID: \"8eb1e02e-a084-43d6-be49-889cc71b330c\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.709184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-root\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-textfile\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709401 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-root\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709401 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-sys\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709401 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-wtmp\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709401 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-sys\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709420 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-textfile\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-wtmp\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:42.709491 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/9111c994-45fe-4cf1-ac20-3f2562da51c6-kube-api-access-gg8fl\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-metrics-client-ca\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709587 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:42.709576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls podName:9111c994-45fe-4cf1-ac20-3f2562da51c6 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:43.209556703 +0000 UTC m=+56.514555636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls") pod "node-exporter-6qhm8" (UID: "9111c994-45fe-4cf1-ac20-3f2562da51c6") : secret "node-exporter-tls" not found Jul 17 21:09:42.709981 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709688 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.709981 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.709974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9111c994-45fe-4cf1-ac20-3f2562da51c6-metrics-client-ca\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.711689 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.711670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.718429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.718412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/9111c994-45fe-4cf1-ac20-3f2562da51c6-kube-api-access-gg8fl\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:42.730296 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.730281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" Jul 17 21:09:42.884310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:42.884281 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr"] Jul 17 21:09:43.112847 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.112816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:43.115227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.115202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/55f622da-ecd7-4413-8629-8cce3056b0ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvd5z\" (UID: \"55f622da-ecd7-4413-8629-8cce3056b0ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:43.213747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.213714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:43.216065 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.216037 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9111c994-45fe-4cf1-ac20-3f2562da51c6-node-exporter-tls\") pod \"node-exporter-6qhm8\" (UID: \"9111c994-45fe-4cf1-ac20-3f2562da51c6\") " pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:43.389557 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.389521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" Jul 17 21:09:43.407649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.407628 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qhm8" Jul 17 21:09:43.417090 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:43.417062 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9111c994_45fe_4cf1_ac20_3f2562da51c6.slice/crio-0e0f473073c4bf772f6971da30c556cab0a903631a04b19d7832eceb6367c220 WatchSource:0}: Error finding container 0e0f473073c4bf772f6971da30c556cab0a903631a04b19d7832eceb6367c220: Status 404 returned error can't find the container with id 0e0f473073c4bf772f6971da30c556cab0a903631a04b19d7832eceb6367c220 Jul 17 21:09:43.529576 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.529544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" event={"ID":"8eb1e02e-a084-43d6-be49-889cc71b330c","Type":"ContainerStarted","Data":"be6f879cfbd9a4c2b5aea82e42e693cb871fd7a3bd62fbde03bcca6b910123a9"} Jul 17 21:09:43.529983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.529600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" event={"ID":"8eb1e02e-a084-43d6-be49-889cc71b330c","Type":"ContainerStarted","Data":"710b123be66a3ad8235b758710e9b4a37e7f4df98a3bb64d10c0453e0305fbfc"} Jul 17 21:09:43.529983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.529626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" event={"ID":"8eb1e02e-a084-43d6-be49-889cc71b330c","Type":"ContainerStarted","Data":"b9187ce3b9a493908c57106989562257a5fcb8b4f21950eb6fb4ed326ee3312f"} Jul 17 21:09:43.530726 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.530703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qhm8" event={"ID":"9111c994-45fe-4cf1-ac20-3f2562da51c6","Type":"ContainerStarted","Data":"0e0f473073c4bf772f6971da30c556cab0a903631a04b19d7832eceb6367c220"} Jul 17 21:09:43.542040 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.542020 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvd5z"] Jul 17 21:09:43.545145 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:43.545115 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f622da_ecd7_4413_8629_8cce3056b0ad.slice/crio-e5d258a0b0cdc626fd83825ba0fac61f9c383d5ac9080b0f4ce53dea49394578 WatchSource:0}: Error finding container e5d258a0b0cdc626fd83825ba0fac61f9c383d5ac9080b0f4ce53dea49394578: Status 404 returned error can't find the container with id e5d258a0b0cdc626fd83825ba0fac61f9c383d5ac9080b0f4ce53dea49394578 Jul 17 21:09:43.546287 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.546253 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:09:43.549509 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.549491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.554596 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.554578 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 21:09:43.556351 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.556330 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 21:09:43.556502 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.556472 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 21:09:43.556601 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.556503 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 21:09:43.557654 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.557635 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 21:09:43.558897 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.558881 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 21:09:43.558897 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.558891 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xngtz\"" Jul 17 21:09:43.559114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.559102 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 21:09:43.559269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.559256 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 21:09:43.559380 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.559359 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 21:09:43.587816 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.587796 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:09:43.617742 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.617948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.617948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.617948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.617948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.617948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhjp\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.617995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.618062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618163 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.618105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618331 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.618174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.618331 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.618216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719449 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719449 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719644 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719700 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719700 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhjp\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.719822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.719790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.720831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.720695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.721899 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.721871 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.722152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.722126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.722980 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.722954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.723227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.723202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.723483 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.723461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.724135 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.724118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.724749 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.724730 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.724922 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.724804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.724922 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.724853 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.725120 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.725100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.726116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.726099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.741734 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.741710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhjp\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp\") pod \"alertmanager-main-0\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:43.859166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:43.859137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:09:44.018682 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:44.018647 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:09:44.169981 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:44.169944 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9af3acf_4507_4352_8796_8628abcd8392.slice/crio-2dd5e91e5357531e583dfa00a69623e2556e4a4fe489907224cd179c57324b6e WatchSource:0}: Error finding container 2dd5e91e5357531e583dfa00a69623e2556e4a4fe489907224cd179c57324b6e: Status 404 returned error can't find the container with id 2dd5e91e5357531e583dfa00a69623e2556e4a4fe489907224cd179c57324b6e Jul 17 21:09:44.535502 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:44.535475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"2dd5e91e5357531e583dfa00a69623e2556e4a4fe489907224cd179c57324b6e"} Jul 17 21:09:44.538151 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:44.538125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" event={"ID":"8eb1e02e-a084-43d6-be49-889cc71b330c","Type":"ContainerStarted","Data":"03cf1b2248a67c2b1f8b5894983fe6b68f84bf126246ce1075a60a82f1a44eb0"} Jul 17 21:09:44.539496 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:44.539473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" event={"ID":"55f622da-ecd7-4413-8629-8cce3056b0ad","Type":"ContainerStarted","Data":"e5d258a0b0cdc626fd83825ba0fac61f9c383d5ac9080b0f4ce53dea49394578"} Jul 17 21:09:44.557536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:44.557495 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-2fvbr" podStartSLOduration=1.3603298449999999 podStartE2EDuration="2.55748238s" totalImagesPullingTime="1.197152535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:44.556340631 +0000 UTC m=+57.861339563" watchObservedRunningTime="2026-07-17 21:09:44.55748238 +0000 UTC m=+57.862481289" Jul 17 21:09:45.107379 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.107356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:45.107508 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.107449 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:45.112531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.112507 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:45.503746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.503715 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg"] Jul 17 21:09:45.506836 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.506810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.510084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510062 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 21:09:45.510230 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510212 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 21:09:45.510356 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510341 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 21:09:45.510452 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510437 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 21:09:45.510530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510516 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 21:09:45.510634 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510619 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnjqr\"" Jul 17 21:09:45.510696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.510681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fk624usp9g9d2\"" Jul 17 21:09:45.518929 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.518880 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg"] Jul 17 21:09:45.536539 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-metrics-client-ca\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-grpc-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkrr\" (UniqueName: \"kubernetes.io/projected/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-kube-api-access-7lkrr\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.536983 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.536958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.548847 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.548818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" event={"ID":"55f622da-ecd7-4413-8629-8cce3056b0ad","Type":"ContainerStarted","Data":"1851a2fbc44ad4186af722bcc9dbd22e1b983ad012417fa0e31c71ed9e7aa6f4"} Jul 17 21:09:45.548966 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.548878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" event={"ID":"55f622da-ecd7-4413-8629-8cce3056b0ad","Type":"ContainerStarted","Data":"ae5895e075f8f06d7e265e2dab386be0aa940bff74fb9e7f4d416ab8faa5ff96"} Jul 17 21:09:45.548966 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.548912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" event={"ID":"55f622da-ecd7-4413-8629-8cce3056b0ad","Type":"ContainerStarted","Data":"f0e8f49f9491083942ac5082c35c1b3202fb224a7fde7242444e105d0d37fd5e"} Jul 17 21:09:45.550961 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.550933 2573 generic.go:356] "Generic (PLEG): container finished" podID="9111c994-45fe-4cf1-ac20-3f2562da51c6" containerID="0cf25b380a3610cd917ee4c03e2c6c5b2d840f5536c57f39fbe53aa6a61bf6b1" exitCode=0 Jul 17 21:09:45.551071 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.551001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qhm8" event={"ID":"9111c994-45fe-4cf1-ac20-3f2562da51c6","Type":"ContainerDied","Data":"0cf25b380a3610cd917ee4c03e2c6c5b2d840f5536c57f39fbe53aa6a61bf6b1"} Jul 17 21:09:45.556054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.556034 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:09:45.567813 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.567756 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvd5z" podStartSLOduration=2.086039137 podStartE2EDuration="3.567742892s" totalImagesPullingTime="1.481703755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:45.5661417 +0000 UTC m=+58.871140633" watchObservedRunningTime="2026-07-17 21:09:45.567742892 +0000 UTC m=+58.872741822" Jul 17 21:09:45.626290 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.626256 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-metrics-client-ca\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-grpc-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkrr\" (UniqueName: \"kubernetes.io/projected/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-kube-api-access-7lkrr\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.638732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.640396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.639721 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-metrics-client-ca\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.650298 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.650248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.650649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.650619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.650755 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.650655 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.650872 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.650855 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.652170 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.652132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.652690 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.652672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-secret-grpc-tls\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.655203 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.655184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkrr\" (UniqueName: \"kubernetes.io/projected/1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e-kube-api-access-7lkrr\") pod \"thanos-querier-5d9cdb648f-kd4hg\" (UID: \"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e\") " pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.817719 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.817695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:45.958059 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:45.958032 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg"] Jul 17 21:09:45.968080 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:45.968056 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df7c130_a6bc_4f63_a54b_9bfe4f41ae2e.slice/crio-7598cc22f1eacb90c8dfb002fdfc6937f5a87658277f0095f34e281059b296e4 WatchSource:0}: Error finding container 7598cc22f1eacb90c8dfb002fdfc6937f5a87658277f0095f34e281059b296e4: Status 404 returned error can't find the container with id 7598cc22f1eacb90c8dfb002fdfc6937f5a87658277f0095f34e281059b296e4 Jul 17 21:09:46.501105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.501076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:09:46.557762 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.557730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qhm8" event={"ID":"9111c994-45fe-4cf1-ac20-3f2562da51c6","Type":"ContainerStarted","Data":"66246de31986c9686d0a5e0e4b845a07db7882ec336c5e01d1ec93f1646c6111"} Jul 17 21:09:46.558161 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.557813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qhm8" event={"ID":"9111c994-45fe-4cf1-ac20-3f2562da51c6","Type":"ContainerStarted","Data":"3fc08c80e28037e0caadacdbd0f2c9438409f2cf747074d3731774b49989af27"} Jul 17 21:09:46.559109 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.559084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"7598cc22f1eacb90c8dfb002fdfc6937f5a87658277f0095f34e281059b296e4"} Jul 17 21:09:46.560687 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.560663 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="31b1ac673a530d3ce0b97016b0d4e6cb69608e894532da59489fe89248d077a1" exitCode=0 Jul 17 21:09:46.560813 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.560756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"31b1ac673a530d3ce0b97016b0d4e6cb69608e894532da59489fe89248d077a1"} Jul 17 21:09:46.577924 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.577863 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6qhm8" podStartSLOduration=3.4469125 podStartE2EDuration="4.57784819s" totalImagesPullingTime="1.054030347s" totalInitContainerRuntime="76.905343ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:46.577333097 +0000 UTC m=+59.882332028" watchObservedRunningTime="2026-07-17 21:09:46.57784819 +0000 UTC m=+59.882847122" Jul 17 21:09:46.899313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.899233 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9"] Jul 17 21:09:46.901979 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.901957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.905193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.905171 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 21:09:46.905578 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.905557 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 21:09:46.905790 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.905753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 21:09:46.905988 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.905962 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cqd67\"" Jul 17 21:09:46.906276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.906255 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-avvcaeankvld7\"" Jul 17 21:09:46.909146 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.906594 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 21:09:46.916154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.916130 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9"] Jul 17 21:09:46.953840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.953818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-client-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954028 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.953856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-client-certs\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954028 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.953910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-metrics-server-audit-profiles\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954028 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.954008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrmw\" (UniqueName: \"kubernetes.io/projected/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-kube-api-access-wbrmw\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.954064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-audit-log\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.954099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:46.954195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:46.954163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-tls\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.054640 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrmw\" (UniqueName: \"kubernetes.io/projected/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-kube-api-access-wbrmw\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.054818 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-audit-log\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.054818 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.054946 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-tls\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.054996 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-client-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.055048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.054997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-client-certs\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.055048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.055013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-audit-log\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.055145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.055108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-metrics-server-audit-profiles\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.057811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.057470 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 21:09:47.057811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.057554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 21:09:47.057811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.057592 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 21:09:47.057811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.057554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-avvcaeankvld7\"" Jul 17 21:09:47.057811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.057624 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 21:09:47.064746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.064725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrmw\" (UniqueName: \"kubernetes.io/projected/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-kube-api-access-wbrmw\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.066855 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.066074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.068064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.068043 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-client-ca-bundle\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.070058 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.070039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-client-certs\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.070266 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.070238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-metrics-server-audit-profiles\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.070437 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.070419 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e26bef88-f99c-4dc0-a53e-ab6f4c22040a-secret-metrics-server-tls\") pod \"metrics-server-7f5c5df4cf-nw5k9\" (UID: \"e26bef88-f99c-4dc0-a53e-ab6f4c22040a\") " pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.219796 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.219688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cqd67\"" Jul 17 21:09:47.227683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.227653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:09:47.228236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.228217 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v"] Jul 17 21:09:47.241479 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.241261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.245418 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.245381 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gls7b\"" Jul 17 21:09:47.246348 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.246219 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 21:09:47.250062 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.250017 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v"] Jul 17 21:09:47.357525 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.357497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.357649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.357569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57h7\" (UniqueName: \"kubernetes.io/projected/164470f9-84d6-41b4-b1be-b867b5f32603-kube-api-access-s57h7\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.413530 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.413464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vb84k" Jul 17 21:09:47.414573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.414550 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9"] Jul 17 21:09:47.414790 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:47.414749 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26bef88_f99c_4dc0_a53e_ab6f4c22040a.slice/crio-20ad89b12a2d5a19508deeb24b11c700cb810cfe6cc39677d5f35f1d197b1ce1 WatchSource:0}: Error finding container 20ad89b12a2d5a19508deeb24b11c700cb810cfe6cc39677d5f35f1d197b1ce1: Status 404 returned error can't find the container with id 20ad89b12a2d5a19508deeb24b11c700cb810cfe6cc39677d5f35f1d197b1ce1 Jul 17 21:09:47.458381 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.458328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.458542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.458400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57h7\" (UniqueName: \"kubernetes.io/projected/164470f9-84d6-41b4-b1be-b867b5f32603-kube-api-access-s57h7\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.458607 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:47.458588 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 21:09:47.458660 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:09:47.458641 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert podName:164470f9-84d6-41b4-b1be-b867b5f32603 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:47.958624229 +0000 UTC m=+61.263623152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-j462v" (UID: "164470f9-84d6-41b4-b1be-b867b5f32603") : secret "monitoring-plugin-cert" not found Jul 17 21:09:47.473644 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.473588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57h7\" (UniqueName: \"kubernetes.io/projected/164470f9-84d6-41b4-b1be-b867b5f32603-kube-api-access-s57h7\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.567717 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.567678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" event={"ID":"e26bef88-f99c-4dc0-a53e-ab6f4c22040a","Type":"ContainerStarted","Data":"20ad89b12a2d5a19508deeb24b11c700cb810cfe6cc39677d5f35f1d197b1ce1"} Jul 17 21:09:47.963003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.962961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:47.966069 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:47.966045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/164470f9-84d6-41b4-b1be-b867b5f32603-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j462v\" (UID: \"164470f9-84d6-41b4-b1be-b867b5f32603\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:48.170739 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.170518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:48.589335 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.589288 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:09:48.612329 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.611764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:09:48.612501 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.612489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669763 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669763 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rjd\" (UniqueName: \"kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669997 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669997 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669997 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.669997 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.669951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.670193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.670056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.733610 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.733575 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:09:48.758037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.756991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:09:48.758037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.757175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.762246 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.762218 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 21:09:48.762424 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.762229 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 21:09:48.762509 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.762469 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 21:09:48.762894 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.762716 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 21:09:48.762894 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.762863 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8g3cri9p3iv29\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763438 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763628 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhn5x\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763664 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763824 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763884 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 21:09:48.764066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.763898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 21:09:48.767475 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.767349 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rjd\" (UniqueName: \"kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.771887 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.771380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.772252 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.772131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.773490 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.772999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.773490 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.773296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.773490 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.773390 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.775161 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.775135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.775631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.775605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.781359 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.781336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rjd\" (UniqueName: \"kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd\") pod \"console-564c8dfb7d-k2hq2\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.872626 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.872750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.872750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcll2\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.872750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.872939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.872939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.872992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.873429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.873352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.924022 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.923988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:48.973890 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.973865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.973972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.973901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.973972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.973929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.973972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.973955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.973980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcll2\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.974729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.974494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.975581 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.976663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.980250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.988268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.989857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.990436 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.991820 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.991104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.992797 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.992742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:48.995950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:48.995915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.005921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.005476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.005921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.005480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.007492 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.007475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.011194 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.011134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.011603 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.011555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.011859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.011835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.016926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.016392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcll2\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.016926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.016672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.017256 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.017059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.020062 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.018976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.020596 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.020520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.061972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.061734 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v"] Jul 17 21:09:49.079473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.078984 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:49.151590 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.151524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:09:49.294032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.293693 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:09:49.579680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.579602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"eb582ca824b2e06b24cd4fe6c95ef5e3f14239b453e5bdd25629daa256ef634f"} Jul 17 21:09:49.579905 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.579721 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"af3e867f1288cee752019cf6e3d2068a84b1f149d3687d481fba8b23d663ea46"} Jul 17 21:09:49.579905 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.579791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"05abb9b14cba8ca50ed6488317d236b88ed80987f76208d6022d1c8c2c02774e"} Jul 17 21:09:49.582058 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.582032 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"d27b5ebe8f52a426340629a2d6c4bbe0e9ba416e7942a6ea9c8294f081561d39"} Jul 17 21:09:49.582177 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.582097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"86104b0417c634fd51eb3e3d37a215a7471914db08518cff3ca6fd30d5c65a5e"} Jul 17 21:09:49.582177 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.582149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"0c20305066c9a125152e171c147450171059606623912a4db246a10f7118d599"} Jul 17 21:09:49.583429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:49.583398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" event={"ID":"164470f9-84d6-41b4-b1be-b867b5f32603","Type":"ContainerStarted","Data":"63149b5e0359ad2cad4732090639d19a32dbd33e06019387043af6618de1e875"} Jul 17 21:09:49.607651 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:49.607605 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e13ad7_3274_49b6_8fb4_8bf19bf320e8.slice/crio-971b6c199bf22d9dab45125af540f2fc009b86a62a81f6566a4e66b77917d2d9 WatchSource:0}: Error finding container 971b6c199bf22d9dab45125af540f2fc009b86a62a81f6566a4e66b77917d2d9: Status 404 returned error can't find the container with id 971b6c199bf22d9dab45125af540f2fc009b86a62a81f6566a4e66b77917d2d9 Jul 17 21:09:49.608288 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:49.608268 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dfa659_61d9_4779_bf48_3c6c538f33d3.slice/crio-0bfc06dc97f7da295367b23d2e0be3c8a7401b4fdea904cfe0dc92ae4facd139 WatchSource:0}: Error finding container 0bfc06dc97f7da295367b23d2e0be3c8a7401b4fdea904cfe0dc92ae4facd139: Status 404 returned error can't find the container with id 0bfc06dc97f7da295367b23d2e0be3c8a7401b4fdea904cfe0dc92ae4facd139 Jul 17 21:09:50.590587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.590554 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" exitCode=0 Jul 17 21:09:50.591017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.590845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} Jul 17 21:09:50.591017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.590938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"0bfc06dc97f7da295367b23d2e0be3c8a7401b4fdea904cfe0dc92ae4facd139"} Jul 17 21:09:50.594764 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.594734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"bd1c232584442315df4bdc6de9b661f5d9dd08a099f448ae141f3086e4f191cb"} Jul 17 21:09:50.594899 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.594823 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"c10d85fba1294f0d99bac6317bc252cf4214132ec9eaee6a6f5472f7a2662cb8"} Jul 17 21:09:50.594899 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.594878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" event={"ID":"1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e","Type":"ContainerStarted","Data":"f84332d83a25260ffd2751d7085619e83d1b51d259fdac9b34a841abd5b9d61a"} Jul 17 21:09:50.595007 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.594914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:50.596673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.596650 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" event={"ID":"e26bef88-f99c-4dc0-a53e-ab6f4c22040a","Type":"ContainerStarted","Data":"23b75cb92bc65dcdb8e587bd74ef9978c343b88a40dadff922711995e2d500ca"} Jul 17 21:09:50.601371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.601340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"cb1e88a544f19a87bad7602ffd8317357c7bc60aad053b523293967abe691672"} Jul 17 21:09:50.601452 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.601405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"3182ec63ab6f2b28fe4807c3017304e869f94837b513e58890f5a795db83b375"} Jul 17 21:09:50.601452 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.601444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerStarted","Data":"13ae41417ee94e71821bd7a040ae4b16e84d0b9c8d5ad198602159985d932c18"} Jul 17 21:09:50.603441 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.603417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564c8dfb7d-k2hq2" event={"ID":"12e13ad7-3274-49b6-8fb4-8bf19bf320e8","Type":"ContainerStarted","Data":"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b"} Jul 17 21:09:50.603528 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.603459 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564c8dfb7d-k2hq2" event={"ID":"12e13ad7-3274-49b6-8fb4-8bf19bf320e8","Type":"ContainerStarted","Data":"971b6c199bf22d9dab45125af540f2fc009b86a62a81f6566a4e66b77917d2d9"} Jul 17 21:09:50.648482 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.648432 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" podStartSLOduration=1.835623829 podStartE2EDuration="5.64841637s" totalImagesPullingTime="3.812792541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:50.64653338 +0000 UTC m=+63.951532308" watchObservedRunningTime="2026-07-17 21:09:50.64841637 +0000 UTC m=+63.953415301" Jul 17 21:09:50.664969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.664902 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" podStartSLOduration=2.440890737 podStartE2EDuration="4.664885575s" totalImagesPullingTime="2.223994838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:50.663911903 +0000 UTC m=+63.968910846" watchObservedRunningTime="2026-07-17 21:09:50.664885575 +0000 UTC m=+63.969884507" Jul 17 21:09:50.699289 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.699232 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.425906491 podStartE2EDuration="7.699212026s" totalImagesPullingTime="4.252965756s" totalInitContainerRuntime="20.339779ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:50.698933232 +0000 UTC m=+64.003932163" watchObservedRunningTime="2026-07-17 21:09:50.699212026 +0000 UTC m=+64.004210958" Jul 17 21:09:50.718675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:50.718623 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-564c8dfb7d-k2hq2" podStartSLOduration=2.718606819 podStartE2EDuration="2.718606819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:50.71667854 +0000 UTC m=+64.021677487" watchObservedRunningTime="2026-07-17 21:09:50.718606819 +0000 UTC m=+64.023605750" Jul 17 21:09:51.609104 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:51.609001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" event={"ID":"164470f9-84d6-41b4-b1be-b867b5f32603","Type":"ContainerStarted","Data":"2de0b4d8df528ffcbb138c573da3e78c8c4d517039e59a892e67f69d1b09ed72"} Jul 17 21:09:51.609641 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:51.609619 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:51.614519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:51.614495 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" Jul 17 21:09:51.625711 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:51.625664 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j462v" podStartSLOduration=2.969226263 podStartE2EDuration="4.625650875s" totalImagesPullingTime="1.656424612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:51.624603525 +0000 UTC m=+64.929602456" watchObservedRunningTime="2026-07-17 21:09:51.625650875 +0000 UTC m=+64.930649837" Jul 17 21:09:53.016786 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.016737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:53.017182 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.016894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:53.019932 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.019886 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:09:53.020672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.020649 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:09:53.029515 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.029490 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92967c53-9967-4227-84fa-408e172b8a27-original-pull-secret\") pod \"global-pull-secret-syncer-k2hd6\" (UID: \"92967c53-9967-4227-84fa-408e172b8a27\") " pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:53.029609 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.029553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ef74bf-c023-43cb-90f1-b8db835926b4-metrics-certs\") pod \"network-metrics-daemon-hnhvt\" (UID: \"20ef74bf-c023-43cb-90f1-b8db835926b4\") " pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:53.118313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.118276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:53.121438 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.121413 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:09:53.131511 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.131493 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:09:53.141467 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.141449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czzz\" (UniqueName: \"kubernetes.io/projected/167e7e67-7a9e-47ad-a05a-7478eedc6602-kube-api-access-9czzz\") pod \"network-check-target-w929x\" (UID: \"167e7e67-7a9e-47ad-a05a-7478eedc6602\") " pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:53.144089 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.144070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\"" Jul 17 21:09:53.149508 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.149487 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\"" Jul 17 21:09:53.151575 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.151557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhvt" Jul 17 21:09:53.151668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.151649 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2hd6" Jul 17 21:09:53.157536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.157514 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:53.792744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.792680 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hnhvt"] Jul 17 21:09:53.799738 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:53.797362 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ef74bf_c023_43cb_90f1_b8db835926b4.slice/crio-4ed70fcb7b1662ec3c7898c91adc50e90efcd229decaf158dd10b30b3e2df3ca WatchSource:0}: Error finding container 4ed70fcb7b1662ec3c7898c91adc50e90efcd229decaf158dd10b30b3e2df3ca: Status 404 returned error can't find the container with id 4ed70fcb7b1662ec3c7898c91adc50e90efcd229decaf158dd10b30b3e2df3ca Jul 17 21:09:53.984620 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.984598 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w929x"] Jul 17 21:09:53.995534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:53.995358 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k2hd6"] Jul 17 21:09:54.000116 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:54.000080 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167e7e67_7a9e_47ad_a05a_7478eedc6602.slice/crio-48eba0d2f1697f979b749e964e335f51553244d9771b731df68e1b0e896d5e3d WatchSource:0}: Error finding container 48eba0d2f1697f979b749e964e335f51553244d9771b731df68e1b0e896d5e3d: Status 404 returned error can't find the container with id 48eba0d2f1697f979b749e964e335f51553244d9771b731df68e1b0e896d5e3d Jul 17 21:09:54.000353 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:09:54.000313 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92967c53_9967_4227_84fa_408e172b8a27.slice/crio-11305ad08c18e565d0436fbe94c35ede17bf18e23e19f9ab5f2d1f8a7b96516d WatchSource:0}: Error finding container 11305ad08c18e565d0436fbe94c35ede17bf18e23e19f9ab5f2d1f8a7b96516d: Status 404 returned error can't find the container with id 11305ad08c18e565d0436fbe94c35ede17bf18e23e19f9ab5f2d1f8a7b96516d Jul 17 21:09:54.627104 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.627066 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k2hd6" event={"ID":"92967c53-9967-4227-84fa-408e172b8a27","Type":"ContainerStarted","Data":"11305ad08c18e565d0436fbe94c35ede17bf18e23e19f9ab5f2d1f8a7b96516d"} Jul 17 21:09:54.629008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.628978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhvt" event={"ID":"20ef74bf-c023-43cb-90f1-b8db835926b4","Type":"ContainerStarted","Data":"4ed70fcb7b1662ec3c7898c91adc50e90efcd229decaf158dd10b30b3e2df3ca"} Jul 17 21:09:54.633077 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} Jul 17 21:09:54.633202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} Jul 17 21:09:54.633202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} Jul 17 21:09:54.633202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633198 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} Jul 17 21:09:54.633354 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} Jul 17 21:09:54.633354 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.633274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerStarted","Data":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} Jul 17 21:09:54.634703 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.634676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w929x" event={"ID":"167e7e67-7a9e-47ad-a05a-7478eedc6602","Type":"ContainerStarted","Data":"48eba0d2f1697f979b749e964e335f51553244d9771b731df68e1b0e896d5e3d"} Jul 17 21:09:54.673078 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:54.665748 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.607925597 podStartE2EDuration="6.665727215s" totalImagesPullingTime="3.028546359s" totalInitContainerRuntime="29.255259ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:54.663036832 +0000 UTC m=+67.968035801" watchObservedRunningTime="2026-07-17 21:09:54.665727215 +0000 UTC m=+67.970726148" Jul 17 21:09:55.643096 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:55.643057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhvt" event={"ID":"20ef74bf-c023-43cb-90f1-b8db835926b4","Type":"ContainerStarted","Data":"ffb5da9b0ce6dfd612a97b50f0f0621de9b8e55f2aeff4e2156ba5c2007de35c"} Jul 17 21:09:55.643462 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:55.643122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhvt" event={"ID":"20ef74bf-c023-43cb-90f1-b8db835926b4","Type":"ContainerStarted","Data":"73db1129899dfb1246b2b473125c252611e27abf2b1d313ae3488850edf132b0"} Jul 17 21:09:55.661682 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:55.661615 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hnhvt" podStartSLOduration=67.485526945 podStartE2EDuration="1m8.661595205s" totalImagesPullingTime="1.17606826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:55.659432208 +0000 UTC m=+68.964431140" watchObservedRunningTime="2026-07-17 21:09:55.661595205 +0000 UTC m=+68.966594161" Jul 17 21:09:56.618451 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:56.618415 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d9cdb648f-kd4hg" Jul 17 21:09:58.924392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:58.924356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:58.924841 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:58.924641 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:58.930847 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:58.930824 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:59.079348 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.079312 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:09:59.676192 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.676116 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w929x" event={"ID":"167e7e67-7a9e-47ad-a05a-7478eedc6602","Type":"ContainerStarted","Data":"3a759ec17cbe619c3f8dd330ad9da767cb765855e76d8fb0688e38d8e14d073c"} Jul 17 21:09:59.676386 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.676234 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:09:59.677645 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.677619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k2hd6" event={"ID":"92967c53-9967-4227-84fa-408e172b8a27","Type":"ContainerStarted","Data":"7fe4a78d0740692ae57f04442def759a474a9009dacbce34911f09d61a2be591"} Jul 17 21:09:59.681611 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.681589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:09:59.699046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.699007 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w929x" podStartSLOduration=67.421019027 podStartE2EDuration="1m12.698995288s" totalImagesPullingTime="5.277976261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:59.697384269 +0000 UTC m=+73.002383200" watchObservedRunningTime="2026-07-17 21:09:59.698995288 +0000 UTC m=+73.003994264" Jul 17 21:09:59.713895 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.713850 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k2hd6" podStartSLOduration=67.424708046 podStartE2EDuration="1m12.713833839s" totalImagesPullingTime="5.289125793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:59.712956111 +0000 UTC m=+73.017955043" watchObservedRunningTime="2026-07-17 21:09:59.713833839 +0000 UTC m=+73.018832771" Jul 17 21:09:59.762148 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:09:59.762123 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:10:07.228478 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:07.228437 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:10:07.228884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:07.228531 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:10:10.653114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:10.652977 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b6b684c8-2wkpw" podUID="84afa984-e4ab-43dd-ba34-26ae129da158" containerName="console" containerID="cri-o://4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94" gracePeriod=15 Jul 17 21:10:11.467948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.467921 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:10:11.593063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.592990 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593033 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593332 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593068 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593332 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593086 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593332 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593172 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqbn\" (UniqueName: \"kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn\") pod \"84afa984-e4ab-43dd-ba34-26ae129da158\" (UID: \"84afa984-e4ab-43dd-ba34-26ae129da158\") " Jul 17 21:10:11.593476 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593428 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca" (OuterVolumeSpecName: "service-ca") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:11.593476 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:11.593476 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.593442 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config" (OuterVolumeSpecName: "console-config") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:11.595536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.595505 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:10:11.595536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.595511 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn" (OuterVolumeSpecName: "kube-api-access-8sqbn") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "kube-api-access-8sqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:10:11.595664 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.595556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "84afa984-e4ab-43dd-ba34-26ae129da158" (UID: "84afa984-e4ab-43dd-ba34-26ae129da158"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:10:11.693811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693764 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.693811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693809 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.693811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693820 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.694217 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693828 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84afa984-e4ab-43dd-ba34-26ae129da158-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.694217 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693836 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84afa984-e4ab-43dd-ba34-26ae129da158-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.694217 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.693844 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqbn\" (UniqueName: \"kubernetes.io/projected/84afa984-e4ab-43dd-ba34-26ae129da158-kube-api-access-8sqbn\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:11.730112 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.730081 2573 generic.go:356] "Generic (PLEG): container finished" podID="84afa984-e4ab-43dd-ba34-26ae129da158" containerID="4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94" exitCode=0 Jul 17 21:10:11.730254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.730125 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6b684c8-2wkpw" Jul 17 21:10:11.730254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.730174 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6b684c8-2wkpw" event={"ID":"84afa984-e4ab-43dd-ba34-26ae129da158","Type":"ContainerDied","Data":"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94"} Jul 17 21:10:11.730254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.730222 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6b684c8-2wkpw" event={"ID":"84afa984-e4ab-43dd-ba34-26ae129da158","Type":"ContainerDied","Data":"f0bacdee39f8684f87919c631fa8837f36becf3ba055a66c8ffce939c80fa98d"} Jul 17 21:10:11.730357 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.730255 2573 scope.go:117] "RemoveContainer" containerID="4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94" Jul 17 21:10:11.741426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.740078 2573 scope.go:117] "RemoveContainer" containerID="4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94" Jul 17 21:10:11.741426 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:10:11.741362 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94\": container with ID starting with 4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94 not found: ID does not exist" containerID="4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94" Jul 17 21:10:11.741426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.741395 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94"} err="failed to get container status \"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94\": rpc error: code = NotFound desc = could not find container \"4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94\": container with ID starting with 4f54f591ae4ae34d85995e6be451e0a76e1a65b29843f98d3122bb8367f15f94 not found: ID does not exist" Jul 17 21:10:11.751195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.751175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:10:11.754606 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:11.754579 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b6b684c8-2wkpw"] Jul 17 21:10:13.237243 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:13.237196 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84afa984-e4ab-43dd-ba34-26ae129da158" path="/var/lib/kubelet/pods/84afa984-e4ab-43dd-ba34-26ae129da158/volumes" Jul 17 21:10:24.783973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:24.783912 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-589dd559dc-t59lk" podUID="2fa78987-1413-4a14-9fde-1e1f335df65b" containerName="console" containerID="cri-o://eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175" gracePeriod=15 Jul 17 21:10:25.019684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.019661 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:10:25.101491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101424 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101466 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101496 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101548 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101625 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm2q2\" (UniqueName: \"kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2\") pod \"2fa78987-1413-4a14-9fde-1e1f335df65b\" (UID: \"2fa78987-1413-4a14-9fde-1e1f335df65b\") " Jul 17 21:10:25.101971 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101947 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config" (OuterVolumeSpecName: "console-config") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:25.101971 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101958 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:25.102052 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101974 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:25.102052 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.101994 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca" (OuterVolumeSpecName: "service-ca") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:10:25.103788 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.103749 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:10:25.103951 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.103929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:10:25.104011 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.103975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2" (OuterVolumeSpecName: "kube-api-access-lm2q2") pod "2fa78987-1413-4a14-9fde-1e1f335df65b" (UID: "2fa78987-1413-4a14-9fde-1e1f335df65b"). InnerVolumeSpecName "kube-api-access-lm2q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203025 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203071 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203088 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203103 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm2q2\" (UniqueName: \"kubernetes.io/projected/2fa78987-1413-4a14-9fde-1e1f335df65b-kube-api-access-lm2q2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203125 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203138 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fa78987-1413-4a14-9fde-1e1f335df65b-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.207017 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.203152 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fa78987-1413-4a14-9fde-1e1f335df65b-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:10:25.789252 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.789218 2573 generic.go:356] "Generic (PLEG): container finished" podID="2fa78987-1413-4a14-9fde-1e1f335df65b" containerID="eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175" exitCode=0 Jul 17 21:10:25.789675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.789277 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589dd559dc-t59lk" Jul 17 21:10:25.789675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.789304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589dd559dc-t59lk" event={"ID":"2fa78987-1413-4a14-9fde-1e1f335df65b","Type":"ContainerDied","Data":"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175"} Jul 17 21:10:25.789675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.789353 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589dd559dc-t59lk" event={"ID":"2fa78987-1413-4a14-9fde-1e1f335df65b","Type":"ContainerDied","Data":"e908fca1c299cc50a92ab69ec2ac2b7064a55232c8e8ab568ac7bf1d70158035"} Jul 17 21:10:25.789675 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.789372 2573 scope.go:117] "RemoveContainer" containerID="eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175" Jul 17 21:10:25.797583 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.797569 2573 scope.go:117] "RemoveContainer" containerID="eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175" Jul 17 21:10:25.797848 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:10:25.797827 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175\": container with ID starting with eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175 not found: ID does not exist" containerID="eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175" Jul 17 21:10:25.797919 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.797858 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175"} err="failed to get container status \"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175\": rpc error: code = NotFound desc = could not find container \"eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175\": container with ID starting with eff39c9206bf1dfa0b87ab65af4ceb18fec01c1ddc4d37859c8a6e5ab44c2175 not found: ID does not exist" Jul 17 21:10:25.804863 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.804845 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:10:25.808719 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:25.808701 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-589dd559dc-t59lk"] Jul 17 21:10:27.237724 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:27.237690 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa78987-1413-4a14-9fde-1e1f335df65b" path="/var/lib/kubelet/pods/2fa78987-1413-4a14-9fde-1e1f335df65b/volumes" Jul 17 21:10:27.238088 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:27.237982 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:10:27.241656 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:27.241631 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f5c5df4cf-nw5k9" Jul 17 21:10:30.684437 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:30.684409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w929x" Jul 17 21:10:49.079622 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:49.079592 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:10:49.101385 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:49.101358 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:10:49.903306 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:10:49.903279 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:02.768588 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.768558 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.768867 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="alertmanager" containerID="cri-o://05abb9b14cba8ca50ed6488317d236b88ed80987f76208d6022d1c8c2c02774e" gracePeriod=120 Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.768922 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-metric" containerID="cri-o://3182ec63ab6f2b28fe4807c3017304e869f94837b513e58890f5a795db83b375" gracePeriod=120 Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.768941 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-web" containerID="cri-o://eb582ca824b2e06b24cd4fe6c95ef5e3f14239b453e5bdd25629daa256ef634f" gracePeriod=120 Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.768968 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy" containerID="cri-o://13ae41417ee94e71821bd7a040ae4b16e84d0b9c8d5ad198602159985d932c18" gracePeriod=120 Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.769037 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="config-reloader" containerID="cri-o://af3e867f1288cee752019cf6e3d2068a84b1f149d3687d481fba8b23d663ea46" gracePeriod=120 Jul 17 21:11:02.769187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.769073 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="prom-label-proxy" containerID="cri-o://cb1e88a544f19a87bad7602ffd8317357c7bc60aad053b523293967abe691672" gracePeriod=120 Jul 17 21:11:02.939958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.939929 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="cb1e88a544f19a87bad7602ffd8317357c7bc60aad053b523293967abe691672" exitCode=0 Jul 17 21:11:02.939958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.939953 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="13ae41417ee94e71821bd7a040ae4b16e84d0b9c8d5ad198602159985d932c18" exitCode=0 Jul 17 21:11:02.939958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.939960 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="af3e867f1288cee752019cf6e3d2068a84b1f149d3687d481fba8b23d663ea46" exitCode=0 Jul 17 21:11:02.939958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.939965 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="05abb9b14cba8ca50ed6488317d236b88ed80987f76208d6022d1c8c2c02774e" exitCode=0 Jul 17 21:11:02.940192 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.939998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"cb1e88a544f19a87bad7602ffd8317357c7bc60aad053b523293967abe691672"} Jul 17 21:11:02.940192 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.940059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"13ae41417ee94e71821bd7a040ae4b16e84d0b9c8d5ad198602159985d932c18"} Jul 17 21:11:02.940192 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.940086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"af3e867f1288cee752019cf6e3d2068a84b1f149d3687d481fba8b23d663ea46"} Jul 17 21:11:02.940192 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:02.940108 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"05abb9b14cba8ca50ed6488317d236b88ed80987f76208d6022d1c8c2c02774e"} Jul 17 21:11:03.949810 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:03.949756 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="3182ec63ab6f2b28fe4807c3017304e869f94837b513e58890f5a795db83b375" exitCode=0 Jul 17 21:11:03.949810 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:03.949807 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9af3acf-4507-4352-8796-8628abcd8392" containerID="eb582ca824b2e06b24cd4fe6c95ef5e3f14239b453e5bdd25629daa256ef634f" exitCode=0 Jul 17 21:11:03.949810 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:03.949803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"3182ec63ab6f2b28fe4807c3017304e869f94837b513e58890f5a795db83b375"} Jul 17 21:11:03.950313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:03.949866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"eb582ca824b2e06b24cd4fe6c95ef5e3f14239b453e5bdd25629daa256ef634f"} Jul 17 21:11:04.008760 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.008737 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:04.136684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136595 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrhjp\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136686 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136751 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136815 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136866 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136892 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.136928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136927 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.137314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.136956 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.137314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.137047 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.137314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.137086 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web\") pod \"b9af3acf-4507-4352-8796-8628abcd8392\" (UID: \"b9af3acf-4507-4352-8796-8628abcd8392\") " Jul 17 21:11:04.137314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.137235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:04.137504 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.137274 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:04.137504 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.137400 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.138245 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.138215 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:11:04.140531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.140496 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp" (OuterVolumeSpecName: "kube-api-access-lrhjp") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "kube-api-access-lrhjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:11:04.140710 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.140678 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.140882 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.140840 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.140999 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.140980 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.141458 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.141431 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:11:04.141559 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.141479 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.141696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.141680 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume" (OuterVolumeSpecName: "config-volume") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.142052 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.142036 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out" (OuterVolumeSpecName: "config-out") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:11:04.144614 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.144594 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.152022 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.151998 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config" (OuterVolumeSpecName: "web-config") pod "b9af3acf-4507-4352-8796-8628abcd8392" (UID: "b9af3acf-4507-4352-8796-8628abcd8392"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:04.238799 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238746 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-config-out\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238799 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238802 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238819 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhjp\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-kube-api-access-lrhjp\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238834 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9af3acf-4507-4352-8796-8628abcd8392-metrics-client-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238845 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-cluster-tls-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238856 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-main-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238865 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238874 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9af3acf-4507-4352-8796-8628abcd8392-tls-assets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238884 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238891 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9af3acf-4507-4352-8796-8628abcd8392-alertmanager-main-db\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238899 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-web-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.238990 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.238907 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9af3acf-4507-4352-8796-8628abcd8392-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:04.958372 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.958330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9af3acf-4507-4352-8796-8628abcd8392","Type":"ContainerDied","Data":"2dd5e91e5357531e583dfa00a69623e2556e4a4fe489907224cd179c57324b6e"} Jul 17 21:11:04.958750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.958389 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:04.958750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.958459 2573 scope.go:117] "RemoveContainer" containerID="cb1e88a544f19a87bad7602ffd8317357c7bc60aad053b523293967abe691672" Jul 17 21:11:04.966725 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.966703 2573 scope.go:117] "RemoveContainer" containerID="3182ec63ab6f2b28fe4807c3017304e869f94837b513e58890f5a795db83b375" Jul 17 21:11:04.973464 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.973448 2573 scope.go:117] "RemoveContainer" containerID="13ae41417ee94e71821bd7a040ae4b16e84d0b9c8d5ad198602159985d932c18" Jul 17 21:11:04.979911 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.979892 2573 scope.go:117] "RemoveContainer" containerID="eb582ca824b2e06b24cd4fe6c95ef5e3f14239b453e5bdd25629daa256ef634f" Jul 17 21:11:04.982580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.982560 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:04.988036 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.988021 2573 scope.go:117] "RemoveContainer" containerID="af3e867f1288cee752019cf6e3d2068a84b1f149d3687d481fba8b23d663ea46" Jul 17 21:11:04.988350 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.988333 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:04.995027 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:04.995012 2573 scope.go:117] "RemoveContainer" containerID="05abb9b14cba8ca50ed6488317d236b88ed80987f76208d6022d1c8c2c02774e" Jul 17 21:11:05.001738 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.001722 2573 scope.go:117] "RemoveContainer" containerID="31b1ac673a530d3ce0b97016b0d4e6cb69608e894532da59489fe89248d077a1" Jul 17 21:11:05.013739 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.013718 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:05.014086 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014068 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014088 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014101 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="prom-label-proxy" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014109 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="prom-label-proxy" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014123 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="init-config-reloader" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014132 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="init-config-reloader" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014142 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84afa984-e4ab-43dd-ba34-26ae129da158" containerName="console" Jul 17 21:11:05.014155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014150 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84afa984-e4ab-43dd-ba34-26ae129da158" containerName="console" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014160 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="alertmanager" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014169 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="alertmanager" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa78987-1413-4a14-9fde-1e1f335df65b" containerName="console" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014205 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa78987-1413-4a14-9fde-1e1f335df65b" containerName="console" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014216 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="config-reloader" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014225 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="config-reloader" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014257 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-web" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014264 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-web" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014273 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-metric" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014289 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-metric" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014357 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="config-reloader" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014369 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-metric" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014380 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84afa984-e4ab-43dd-ba34-26ae129da158" containerName="console" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014389 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy-web" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014399 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="kube-rbac-proxy" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014409 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="alertmanager" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014419 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af3acf-4507-4352-8796-8628abcd8392" containerName="prom-label-proxy" Jul 17 21:11:05.014491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.014430 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa78987-1413-4a14-9fde-1e1f335df65b" containerName="console" Jul 17 21:11:05.021873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.021854 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.024445 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xngtz\"" Jul 17 21:11:05.024542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024482 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 21:11:05.024542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024500 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 21:11:05.024727 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024712 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 21:11:05.024727 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024718 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 21:11:05.024879 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 21:11:05.024931 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024893 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 21:11:05.024982 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.024955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 21:11:05.025035 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.025011 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 21:11:05.029932 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.029912 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 21:11:05.032175 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.032144 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:05.145954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.145877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.145954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.145913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.145954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.145935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxmr\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-kube-api-access-8jxmr\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146484 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146484 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146484 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.146484 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.146357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.237538 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.237500 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9af3acf-4507-4352-8796-8628abcd8392" path="/var/lib/kubelet/pods/b9af3acf-4507-4352-8796-8628abcd8392/volumes" Jul 17 21:11:05.246696 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.246676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.246840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.246707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.246840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.246726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.246840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.246820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247016 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.246866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247016 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxmr\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-kube-api-access-8jxmr\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247713 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.247897 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.247874 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.248188 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.248162 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250115 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.249849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250115 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.249972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250115 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.250057 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250365 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.250137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.250476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250801 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.250758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.250801 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.250792 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.251066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.251045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.251720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.251701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.256447 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.256421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxmr\" (UniqueName: \"kubernetes.io/projected/ba79ee40-7173-4759-bbd8-6d6a1113b2bb-kube-api-access-8jxmr\") pod \"alertmanager-main-0\" (UID: \"ba79ee40-7173-4759-bbd8-6d6a1113b2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.334403 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.334361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:11:05.480096 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.480071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:11:05.481110 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:11:05.481088 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba79ee40_7173_4759_bbd8_6d6a1113b2bb.slice/crio-720080ebee94ecc7c839f0ec0f306ce714be6cd27526177f8df4e64b564ddfe9 WatchSource:0}: Error finding container 720080ebee94ecc7c839f0ec0f306ce714be6cd27526177f8df4e64b564ddfe9: Status 404 returned error can't find the container with id 720080ebee94ecc7c839f0ec0f306ce714be6cd27526177f8df4e64b564ddfe9 Jul 17 21:11:05.964180 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.964144 2573 generic.go:356] "Generic (PLEG): container finished" podID="ba79ee40-7173-4759-bbd8-6d6a1113b2bb" containerID="72ed56a8dea489f578ac438f2b686bed69e1d723f847e93cbf1ebbb47ad78e3a" exitCode=0 Jul 17 21:11:05.964542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.964219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerDied","Data":"72ed56a8dea489f578ac438f2b686bed69e1d723f847e93cbf1ebbb47ad78e3a"} Jul 17 21:11:05.964542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:05.964299 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"720080ebee94ecc7c839f0ec0f306ce714be6cd27526177f8df4e64b564ddfe9"} Jul 17 21:11:06.819712 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.819672 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56555c9968-24q7c"] Jul 17 21:11:06.822529 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.822504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.825551 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.825523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 21:11:06.825684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.825560 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 21:11:06.825760 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.825680 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 21:11:06.825877 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.825822 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 21:11:06.825968 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.825953 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 21:11:06.826759 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.826743 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-csvmh\"" Jul 17 21:11:06.831584 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.831557 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 21:11:06.839305 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.839278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56555c9968-24q7c"] Jul 17 21:11:06.963931 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.963816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.963931 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.963886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.963975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-serving-certs-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.964013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.964045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.964082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgq45\" (UniqueName: \"kubernetes.io/projected/07fa8142-e2c2-4d8d-8257-2d9899ac2418-kube-api-access-cgq45\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.964155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-federate-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.964195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.964183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-metrics-client-ca\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:06.973420 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973388 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"4773ec7beadb4cbdea855de2488fa392be56f9b9194b96481be0f6e7c00efce6"} Jul 17 21:11:06.973574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973448 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"e0bc9d751ed3b52fd57d7d8ec0ca93753fcbd429f5b25f19d372145038deeb9e"} Jul 17 21:11:06.973574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"eebdd9d632b11a017b8f2a08435328d6ba450b96702f6fc1b9a862aa9cc29c67"} Jul 17 21:11:06.973574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973506 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"f5acd4436f2456400ac148b47df5942b32789620a4ab8d136539838c68de36b4"} Jul 17 21:11:06.973574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"b8bc70db49bf997b228b8c5e4f4a268a2f1f764d2bab06a1a8a283e5e7fa4bbe"} Jul 17 21:11:06.973574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:06.973554 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba79ee40-7173-4759-bbd8-6d6a1113b2bb","Type":"ContainerStarted","Data":"6fd41ce465d9d5933d915550e9f9316d35052a8112937aa9eb779eef56cf839b"} Jul 17 21:11:07.007968 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.007902 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.989816774 podStartE2EDuration="3.007875877s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.059103ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:11:07.005121752 +0000 UTC m=+140.310120684" watchObservedRunningTime="2026-07-17 21:11:07.007875877 +0000 UTC m=+140.312874809" Jul 17 21:11:07.065213 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065394 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065394 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-serving-certs-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgq45\" (UniqueName: \"kubernetes.io/projected/07fa8142-e2c2-4d8d-8257-2d9899ac2418-kube-api-access-cgq45\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-federate-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.065683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.065576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-metrics-client-ca\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.066373 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.066286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.066373 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.066307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-metrics-client-ca\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.066373 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.066316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fa8142-e2c2-4d8d-8257-2d9899ac2418-serving-certs-ca-bundle\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.068260 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.068236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-telemeter-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.068582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.068559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.068673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.068581 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.068829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.068808 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07fa8142-e2c2-4d8d-8257-2d9899ac2418-federate-client-tls\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.086731 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.086678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgq45\" (UniqueName: \"kubernetes.io/projected/07fa8142-e2c2-4d8d-8257-2d9899ac2418-kube-api-access-cgq45\") pod \"telemeter-client-56555c9968-24q7c\" (UID: \"07fa8142-e2c2-4d8d-8257-2d9899ac2418\") " pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.146799 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.146747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" Jul 17 21:11:07.151569 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151540 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151845 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="prometheus" containerID="cri-o://2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" gracePeriod=600 Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151875 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy" containerID="cri-o://0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" gracePeriod=600 Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151901 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="config-reloader" containerID="cri-o://50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" gracePeriod=600 Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151900 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="thanos-sidecar" containerID="cri-o://b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" gracePeriod=600 Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151929 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-web" containerID="cri-o://8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" gracePeriod=600 Jul 17 21:11:07.152032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.151970 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" gracePeriod=600 Jul 17 21:11:07.324577 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.324553 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56555c9968-24q7c"] Jul 17 21:11:07.326446 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:11:07.326413 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fa8142_e2c2_4d8d_8257_2d9899ac2418.slice/crio-b9d991899472866df62dc5bfec9708c8701e627c9cdb3745a917a1208bf749e7 WatchSource:0}: Error finding container b9d991899472866df62dc5bfec9708c8701e627c9cdb3745a917a1208bf749e7: Status 404 returned error can't find the container with id b9d991899472866df62dc5bfec9708c8701e627c9cdb3745a917a1208bf749e7 Jul 17 21:11:07.411638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.411611 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:07.573135 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573096 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573135 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573132 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573152 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573172 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573219 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573627 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573650 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573726 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573691 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573799 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573745 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcll2\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573796 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573821 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.573859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.573918 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574125 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574103 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574178 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574144 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574229 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574287 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574226 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.574287 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574253 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.574380 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle\") pod \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\" (UID: \"a6dfa659-61d9-4779-bf48-3c6c538f33d3\") " Jul 17 21:11:07.575873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574647 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.575873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.574989 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.575873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.575237 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.575873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.575282 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.575873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.575744 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.576237 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.576237 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576182 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.576334 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576280 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:07.576495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576435 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.576618 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576547 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.576751 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576705 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.576928 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.576908 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config" (OuterVolumeSpecName: "config") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.577162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.577138 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:11:07.578298 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.578256 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.578432 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.578397 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.578746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.578711 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out" (OuterVolumeSpecName: "config-out") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:11:07.579021 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.579003 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:11:07.579186 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.579169 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2" (OuterVolumeSpecName: "kube-api-access-jcll2") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "kube-api-access-jcll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:11:07.579582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.579564 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.589255 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.589236 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config" (OuterVolumeSpecName: "web-config") pod "a6dfa659-61d9-4779-bf48-3c6c538f33d3" (UID: "a6dfa659-61d9-4779-bf48-3c6c538f33d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675498 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcll2\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-kube-api-access-jcll2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675531 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-metrics-client-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675544 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675558 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-db\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675569 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675582 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dfa659-61d9-4779-bf48-3c6c538f33d3-tls-assets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675595 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-web-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675606 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675619 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675633 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675644 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675655 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-kube-rbac-proxy\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675666 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675679 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675690 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-grpc-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675702 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-metrics-client-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675712 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dfa659-61d9-4779-bf48-3c6c538f33d3-config-out\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675723 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dfa659-61d9-4779-bf48-3c6c538f33d3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.675970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.675735 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6dfa659-61d9-4779-bf48-3c6c538f33d3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983473 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" exitCode=0 Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983500 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" exitCode=0 Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983507 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" exitCode=0 Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983512 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" exitCode=0 Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983517 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" exitCode=0 Jul 17 21:11:07.983560 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983522 2573 generic.go:356] "Generic (PLEG): container finished" podID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" exitCode=0 Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983637 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983735 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983752 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} Jul 17 21:11:07.984108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.983894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6dfa659-61d9-4779-bf48-3c6c538f33d3","Type":"ContainerDied","Data":"0bfc06dc97f7da295367b23d2e0be3c8a7401b4fdea904cfe0dc92ae4facd139"} Jul 17 21:11:07.985426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.985395 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" event={"ID":"07fa8142-e2c2-4d8d-8257-2d9899ac2418","Type":"ContainerStarted","Data":"b9d991899472866df62dc5bfec9708c8701e627c9cdb3745a917a1208bf749e7"} Jul 17 21:11:07.993352 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:07.993334 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.002240 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.002218 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.012456 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.012406 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.013825 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.013765 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:08.021063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.021041 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:08.021844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.021825 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.029186 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.029172 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.039643 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.039623 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.047421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.047388 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.047667 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.047643 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.047749 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.047677 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.047749 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.047704 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.047959 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.047942 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.048003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.047965 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.048003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.047981 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.048264 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.048241 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.048342 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.048272 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.048342 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.048295 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.048564 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.048546 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.048646 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.048572 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.048646 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.048592 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.049122 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.049103 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.049200 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049131 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.049200 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049144 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.049428 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.049408 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.049525 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049432 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.049525 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049447 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.049796 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:08.049659 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.049796 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049689 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.049796 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.049710 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.050029 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050009 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.050114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050030 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.050372 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050272 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.050372 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050308 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.050597 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050579 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.050650 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050599 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.050866 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050841 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.050866 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.050867 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.051105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051087 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.051105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051107 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.051365 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051346 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.051365 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051365 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.051599 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051581 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.051673 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051602 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.051870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051850 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.051870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.051869 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.052139 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.052114 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.052210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.052141 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.052546 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.052450 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.052546 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.052486 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.054248 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054220 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.054248 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054250 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.054582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054560 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.054685 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054589 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.054954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054862 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.054954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054890 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.054954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.054929 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:08.055130 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055081 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.055130 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055100 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.055346 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055330 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="thanos-sidecar" Jul 17 21:11:08.055403 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055352 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="thanos-sidecar" Jul 17 21:11:08.055403 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055353 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.055403 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055367 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="init-config-reloader" Jul 17 21:11:08.055403 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055374 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055376 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="init-config-reloader" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055424 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055431 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055440 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="prometheus" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055446 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="prometheus" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055457 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="config-reloader" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055465 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="config-reloader" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055484 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-web" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055492 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-web" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055504 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-thanos" Jul 17 21:11:08.055536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055512 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-thanos" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055606 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-web" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055619 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="thanos-sidecar" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055632 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy-thanos" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055643 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="prometheus" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055642 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055653 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="kube-rbac-proxy" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055657 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055664 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" containerName="config-reloader" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055918 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.056001 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.055933 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.056369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.056369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056239 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.056561 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056539 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.056617 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056564 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.056840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056817 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.056840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.056839 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.057109 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057084 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.057198 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057110 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.057386 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057360 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.057386 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057386 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.057619 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057596 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.057619 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057617 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.057877 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057857 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.057962 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.057884 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.058134 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058113 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.058213 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058135 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.058411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058386 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.058411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058411 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.058873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058743 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.058873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.058787 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.059046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059014 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.059046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059040 2573 scope.go:117] "RemoveContainer" containerID="b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4" Jul 17 21:11:08.059335 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059287 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4"} err="failed to get container status \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": rpc error: code = NotFound desc = could not find container \"b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4\": container with ID starting with b85022bba3987981071a55df5bec13cedf10c8ef95c24409a9e4d2c16a8ce9b4 not found: ID does not exist" Jul 17 21:11:08.059335 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059313 2573 scope.go:117] "RemoveContainer" containerID="0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395" Jul 17 21:11:08.059566 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059548 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395"} err="failed to get container status \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": rpc error: code = NotFound desc = could not find container \"0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395\": container with ID starting with 0b7adbbb14cd0e26029472e23c4448dc843027a96b0a5f2e5313accd91e07395 not found: ID does not exist" Jul 17 21:11:08.059566 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059566 2573 scope.go:117] "RemoveContainer" containerID="8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1" Jul 17 21:11:08.059991 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059966 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1"} err="failed to get container status \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": rpc error: code = NotFound desc = could not find container \"8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1\": container with ID starting with 8b92494ac181311c4d3f635d1a2580a2bcf77e1940c83ab5c211cb08b53a3cc1 not found: ID does not exist" Jul 17 21:11:08.060088 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.059992 2573 scope.go:117] "RemoveContainer" containerID="b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c" Jul 17 21:11:08.060141 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.060108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.061861 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.061694 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c"} err="failed to get container status \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": rpc error: code = NotFound desc = could not find container \"b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c\": container with ID starting with b46983bf726038250690880cd536b133456ac87f1f48078bb34ec96e5ef7477c not found: ID does not exist" Jul 17 21:11:08.061861 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.061719 2573 scope.go:117] "RemoveContainer" containerID="50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52" Jul 17 21:11:08.064281 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064070 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52"} err="failed to get container status \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": rpc error: code = NotFound desc = could not find container \"50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52\": container with ID starting with 50284bff4c949d07dcca706bfc1dab0643734dd7825cd63c37ae0da666266d52 not found: ID does not exist" Jul 17 21:11:08.064281 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064095 2573 scope.go:117] "RemoveContainer" containerID="2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376" Jul 17 21:11:08.064454 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064276 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064625 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376"} err="failed to get container status \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": rpc error: code = NotFound desc = could not find container \"2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376\": container with ID starting with 2f4fa98a64ce1a942e5b086d97fb296a28740c6f39b4af875b46b5b67dc0d376 not found: ID does not exist" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064686 2573 scope.go:117] "RemoveContainer" containerID="8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064697 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064789 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064853 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 21:11:08.065019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.064940 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b"} err="failed to get container status \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": rpc error: code = NotFound desc = could not find container \"8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b\": container with ID starting with 8f719153a9475f52d96f17d8d7af3caa2e886fa80aefec08e7a7f883a5893c0b not found: ID does not exist" Jul 17 21:11:08.065522 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065088 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 21:11:08.065522 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065109 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhn5x\"" Jul 17 21:11:08.065522 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065145 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 21:11:08.065522 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065276 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 21:11:08.065728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065590 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 21:11:08.065728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 21:11:08.065840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065826 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 21:11:08.065979 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.065962 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8g3cri9p3iv29\"" Jul 17 21:11:08.066157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.066133 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 21:11:08.066736 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.066721 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 21:11:08.070690 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.070131 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 21:11:08.076613 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.076453 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 21:11:08.077649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.077629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:08.180188 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180374 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180745 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.180994 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.180970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181148 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcsf9\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-kube-api-access-vcsf9\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181437 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181586 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181586 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.181720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.181714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282430 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282430 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcsf9\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-kube-api-access-vcsf9\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.282667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.283003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.283510 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.283363 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.284028 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.283628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.284028 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.283732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.284853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.284518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.284853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.282870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.287263 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.286242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.287263 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.286948 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.287263 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.287029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.287463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.287284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.288433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.287582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.288433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.287762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.288433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.288145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.288433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.288296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.288972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.288554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.289816 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.289630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.290066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.290025 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.290066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.290055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.290616 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.290593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.292345 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.292322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.292422 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.292405 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcsf9\" (UniqueName: \"kubernetes.io/projected/d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38-kube-api-access-vcsf9\") pod \"prometheus-k8s-0\" (UID: \"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.375009 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.374969 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:08.525166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.525135 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:11:08.528864 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:11:08.528835 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2662cc6_0b16_46cb_8bb6_3d36ff6f9b38.slice/crio-02753197fdfca456b364b8d11a39ae3206d5b2c55b3769942da16b6c9f6cd08e WatchSource:0}: Error finding container 02753197fdfca456b364b8d11a39ae3206d5b2c55b3769942da16b6c9f6cd08e: Status 404 returned error can't find the container with id 02753197fdfca456b364b8d11a39ae3206d5b2c55b3769942da16b6c9f6cd08e Jul 17 21:11:08.994073 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.994044 2573 generic.go:356] "Generic (PLEG): container finished" podID="d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38" containerID="c8758708bb0641d053226d5fa2723a96322b551d1fba7d5141465acd0ff941b2" exitCode=0 Jul 17 21:11:08.994454 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.994096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerDied","Data":"c8758708bb0641d053226d5fa2723a96322b551d1fba7d5141465acd0ff941b2"} Jul 17 21:11:08.994454 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:08.994146 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"02753197fdfca456b364b8d11a39ae3206d5b2c55b3769942da16b6c9f6cd08e"} Jul 17 21:11:09.237356 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:09.237328 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6dfa659-61d9-4779-bf48-3c6c538f33d3" path="/var/lib/kubelet/pods/a6dfa659-61d9-4779-bf48-3c6c538f33d3/volumes" Jul 17 21:11:10.000184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.000151 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" event={"ID":"07fa8142-e2c2-4d8d-8257-2d9899ac2418","Type":"ContainerStarted","Data":"1ffa9a1292ab1d507fc99ec409f0ef67b5866b1490adbc5b775833c4630b8434"} Jul 17 21:11:10.000630 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.000244 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" event={"ID":"07fa8142-e2c2-4d8d-8257-2d9899ac2418","Type":"ContainerStarted","Data":"4f4a86ce033c7df0d8fe6f28748ecdcc92302af1753d987ba9f309522606d79b"} Jul 17 21:11:10.000630 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.000276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" event={"ID":"07fa8142-e2c2-4d8d-8257-2d9899ac2418","Type":"ContainerStarted","Data":"c33a04a63613adfe69a11e3eec76c723045c9b7a37957e6b13dffde961ef1969"} Jul 17 21:11:10.003164 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"7403d45c64e2ed7ef76cc2cc0f1b3913bb566491542d0940cee02faf9f09c09a"} Jul 17 21:11:10.003236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003204 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"5c450016abf4b02c637e89b828a84a0e1fffcfd25eeb5dcc21eba69c79e7460d"} Jul 17 21:11:10.003236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"899bb9d4c82b1092b0d59fdd1d20195b3da679ffb23fde8d75ad2b5d05657fa5"} Jul 17 21:11:10.003309 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"275b40329c015bb22f0812cffc9676fa119ac2a143c9f798c2ca75a27522fa5f"} Jul 17 21:11:10.003309 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003296 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"c3299c8c6d498335215662adebae90c8de54f3366430b9a238ad76998c4b9eab"} Jul 17 21:11:10.003370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.003323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38","Type":"ContainerStarted","Data":"5308cb96f4457dfa65c6f6c9f882d454609d867c4d9fd974ea719d7c1a1abbef"} Jul 17 21:11:10.023380 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.023338 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56555c9968-24q7c" podStartSLOduration=2.187920988 podStartE2EDuration="4.023324725s" totalImagesPullingTime="1.835403737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:11:10.022367079 +0000 UTC m=+143.327366036" watchObservedRunningTime="2026-07-17 21:11:10.023324725 +0000 UTC m=+143.328323657" Jul 17 21:11:10.050950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.050896 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.024017975 podStartE2EDuration="2.050880009s" totalImagesPullingTime="0s" totalInitContainerRuntime="26.862034ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:11:10.050630162 +0000 UTC m=+143.355629120" watchObservedRunningTime="2026-07-17 21:11:10.050880009 +0000 UTC m=+143.355878939" Jul 17 21:11:10.615677 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.615645 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:11:10.619157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.619138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.631053 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.631033 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:11:10.703112 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7kb\" (UniqueName: \"kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703276 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703409 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.703409 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.703337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.804616 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.804806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.804806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.804806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7kb\" (UniqueName: \"kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.804806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.805023 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.805023 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.804853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.805454 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.805425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.806252 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.806223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.806470 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.806451 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.806636 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.806619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.808508 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.808487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.809090 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.809069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.813429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.813409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7kb\" (UniqueName: \"kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb\") pod \"console-5447fd58f4-slhrh\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:10.928934 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:10.928850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:11.079006 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:11.078966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:11:11.082499 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:11:11.082475 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1aa69a_452e_4708_9c30_cabbb1910c74.slice/crio-e0c0e3cd1dea4478841f510737dc37c69faddadee24da56b88662c5ec149caa3 WatchSource:0}: Error finding container e0c0e3cd1dea4478841f510737dc37c69faddadee24da56b88662c5ec149caa3: Status 404 returned error can't find the container with id e0c0e3cd1dea4478841f510737dc37c69faddadee24da56b88662c5ec149caa3 Jul 17 21:11:12.013966 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:12.013931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5447fd58f4-slhrh" event={"ID":"6c1aa69a-452e-4708-9c30-cabbb1910c74","Type":"ContainerStarted","Data":"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e"} Jul 17 21:11:12.014115 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:12.014006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5447fd58f4-slhrh" event={"ID":"6c1aa69a-452e-4708-9c30-cabbb1910c74","Type":"ContainerStarted","Data":"e0c0e3cd1dea4478841f510737dc37c69faddadee24da56b88662c5ec149caa3"} Jul 17 21:11:12.032188 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:12.032135 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5447fd58f4-slhrh" podStartSLOduration=2.03211997 podStartE2EDuration="2.03211997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:11:12.030503431 +0000 UTC m=+145.335502377" watchObservedRunningTime="2026-07-17 21:11:12.03211997 +0000 UTC m=+145.337118979" Jul 17 21:11:13.375653 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:13.375620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:11:20.929023 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:20.928983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:20.929460 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:20.929073 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:20.933757 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:20.933732 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:21.060533 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:21.060498 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:11:21.106886 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:21.106858 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:11:46.132190 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.132063 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-564c8dfb7d-k2hq2" podUID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" containerName="console" containerID="cri-o://0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b" gracePeriod=15 Jul 17 21:11:46.366821 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.366798 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:11:46.530031 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530003 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530189 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530189 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530112 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530189 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530189 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530174 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7rjd\" (UniqueName: \"kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530347 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530204 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530347 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530236 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert\") pod \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\" (UID: \"12e13ad7-3274-49b6-8fb4-8bf19bf320e8\") " Jul 17 21:11:46.530548 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530523 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:46.530662 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530539 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:46.530662 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:46.530662 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.530641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config" (OuterVolumeSpecName: "console-config") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:11:46.532540 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.532507 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:46.532631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.532536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd" (OuterVolumeSpecName: "kube-api-access-n7rjd") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "kube-api-access-n7rjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:11:46.532631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.532566 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "12e13ad7-3274-49b6-8fb4-8bf19bf320e8" (UID: "12e13ad7-3274-49b6-8fb4-8bf19bf320e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:11:46.631494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631469 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631492 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631500 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631510 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631518 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rjd\" (UniqueName: \"kubernetes.io/projected/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-kube-api-access-n7rjd\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631527 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:46.631668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:46.631536 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12e13ad7-3274-49b6-8fb4-8bf19bf320e8-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:11:47.163246 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.163217 2573 generic.go:356] "Generic (PLEG): container finished" podID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" containerID="0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b" exitCode=0 Jul 17 21:11:47.163604 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.163264 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564c8dfb7d-k2hq2" Jul 17 21:11:47.163604 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.163309 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564c8dfb7d-k2hq2" event={"ID":"12e13ad7-3274-49b6-8fb4-8bf19bf320e8","Type":"ContainerDied","Data":"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b"} Jul 17 21:11:47.163604 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.163372 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564c8dfb7d-k2hq2" event={"ID":"12e13ad7-3274-49b6-8fb4-8bf19bf320e8","Type":"ContainerDied","Data":"971b6c199bf22d9dab45125af540f2fc009b86a62a81f6566a4e66b77917d2d9"} Jul 17 21:11:47.163604 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.163406 2573 scope.go:117] "RemoveContainer" containerID="0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b" Jul 17 21:11:47.172053 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.172034 2573 scope.go:117] "RemoveContainer" containerID="0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b" Jul 17 21:11:47.172521 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:11:47.172500 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b\": container with ID starting with 0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b not found: ID does not exist" containerID="0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b" Jul 17 21:11:47.172587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.172531 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b"} err="failed to get container status \"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b\": rpc error: code = NotFound desc = could not find container \"0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b\": container with ID starting with 0431b68cafcb3902d02f3c1a299101a8b5f32851371d548d2e17829f0c885d3b not found: ID does not exist" Jul 17 21:11:47.191234 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.191216 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:11:47.200796 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.200748 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564c8dfb7d-k2hq2"] Jul 17 21:11:47.236903 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:11:47.236876 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" path="/var/lib/kubelet/pods/12e13ad7-3274-49b6-8fb4-8bf19bf320e8/volumes" Jul 17 21:12:08.375513 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:08.375475 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:12:08.391344 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:08.391320 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:12:09.266443 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:09.266417 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:12:26.831914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.831883 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:12:26.832357 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.832240 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" containerName="console" Jul 17 21:12:26.832357 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.832255 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" containerName="console" Jul 17 21:12:26.832357 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.832319 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e13ad7-3274-49b6-8fb4-8bf19bf320e8" containerName="console" Jul 17 21:12:26.835553 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.835535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.849650 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.849621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:12:26.983746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgld\" (UniqueName: \"kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.983746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.984024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.984024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.984024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.984024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:26.984024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:26.983933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgld\" (UniqueName: \"kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084693 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084693 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.084693 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.084680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.085359 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.085305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.085487 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.085369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.085487 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.085386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.085487 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.085375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.087074 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.087047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.087176 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.087136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.093629 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.093611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgld\" (UniqueName: \"kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld\") pod \"console-7d4b469b68-hzcjj\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.144594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.144566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:27.282239 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.282217 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:12:27.284500 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:12:27.284459 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7d6fff_808d_4af6_9eb1_d868659792f9.slice/crio-89f0d3c4cce1e8f18d59778155df58d4ad24e940028e4b1ccc5eb9ffe01bb2a7 WatchSource:0}: Error finding container 89f0d3c4cce1e8f18d59778155df58d4ad24e940028e4b1ccc5eb9ffe01bb2a7: Status 404 returned error can't find the container with id 89f0d3c4cce1e8f18d59778155df58d4ad24e940028e4b1ccc5eb9ffe01bb2a7 Jul 17 21:12:27.320052 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:27.320018 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b469b68-hzcjj" event={"ID":"da7d6fff-808d-4af6-9eb1-d868659792f9","Type":"ContainerStarted","Data":"89f0d3c4cce1e8f18d59778155df58d4ad24e940028e4b1ccc5eb9ffe01bb2a7"} Jul 17 21:12:28.325793 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:28.325746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b469b68-hzcjj" event={"ID":"da7d6fff-808d-4af6-9eb1-d868659792f9","Type":"ContainerStarted","Data":"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000"} Jul 17 21:12:28.346366 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:28.346320 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4b469b68-hzcjj" podStartSLOduration=2.346304425 podStartE2EDuration="2.346304425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:12:28.344699773 +0000 UTC m=+221.649698704" watchObservedRunningTime="2026-07-17 21:12:28.346304425 +0000 UTC m=+221.651303355" Jul 17 21:12:37.145572 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:37.145539 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:37.146054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:37.145593 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:37.150379 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:37.150357 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:37.364604 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:37.364576 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:12:37.415333 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:12:37.415254 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:13:02.436710 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.436661 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5447fd58f4-slhrh" podUID="6c1aa69a-452e-4708-9c30-cabbb1910c74" containerName="console" containerID="cri-o://29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e" gracePeriod=15 Jul 17 21:13:02.672562 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.672535 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:13:02.795531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795490 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx7kb\" (UniqueName: \"kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795805 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795565 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795805 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795620 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795805 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795643 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795805 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795669 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.795805 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.795723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert\") pod \"6c1aa69a-452e-4708-9c30-cabbb1910c74\" (UID: \"6c1aa69a-452e-4708-9c30-cabbb1910c74\") " Jul 17 21:13:02.796069 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.796041 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca" (OuterVolumeSpecName: "service-ca") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:13:02.796069 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.796050 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config" (OuterVolumeSpecName: "console-config") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:13:02.796182 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.796103 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:13:02.796298 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.796276 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:13:02.797856 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.797832 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb" (OuterVolumeSpecName: "kube-api-access-lx7kb") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "kube-api-access-lx7kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:13:02.798010 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.797992 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:13:02.798051 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.798031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6c1aa69a-452e-4708-9c30-cabbb1910c74" (UID: "6c1aa69a-452e-4708-9c30-cabbb1910c74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:13:02.896794 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896739 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7kb\" (UniqueName: \"kubernetes.io/projected/6c1aa69a-452e-4708-9c30-cabbb1910c74-kube-api-access-lx7kb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896803 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896819 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896831 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896843 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896855 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c1aa69a-452e-4708-9c30-cabbb1910c74-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:02.896952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:02.896864 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c1aa69a-452e-4708-9c30-cabbb1910c74-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:13:03.465496 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.465463 2573 generic.go:356] "Generic (PLEG): container finished" podID="6c1aa69a-452e-4708-9c30-cabbb1910c74" containerID="29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e" exitCode=0 Jul 17 21:13:03.465914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.465515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5447fd58f4-slhrh" event={"ID":"6c1aa69a-452e-4708-9c30-cabbb1910c74","Type":"ContainerDied","Data":"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e"} Jul 17 21:13:03.465914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.465520 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5447fd58f4-slhrh" Jul 17 21:13:03.465914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.465553 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5447fd58f4-slhrh" event={"ID":"6c1aa69a-452e-4708-9c30-cabbb1910c74","Type":"ContainerDied","Data":"e0c0e3cd1dea4478841f510737dc37c69faddadee24da56b88662c5ec149caa3"} Jul 17 21:13:03.465914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.465573 2573 scope.go:117] "RemoveContainer" containerID="29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e" Jul 17 21:13:03.473968 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.473950 2573 scope.go:117] "RemoveContainer" containerID="29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e" Jul 17 21:13:03.474239 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:13:03.474220 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e\": container with ID starting with 29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e not found: ID does not exist" containerID="29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e" Jul 17 21:13:03.474300 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.474248 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e"} err="failed to get container status \"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e\": rpc error: code = NotFound desc = could not find container \"29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e\": container with ID starting with 29997d6ed1ffa3c0bd7b86d4941e86f903cebba373cb5029627c9ebdea043e0e not found: ID does not exist" Jul 17 21:13:03.482071 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.482045 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:13:03.485539 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:03.485518 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5447fd58f4-slhrh"] Jul 17 21:13:05.238357 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:05.238321 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1aa69a-452e-4708-9c30-cabbb1910c74" path="/var/lib/kubelet/pods/6c1aa69a-452e-4708-9c30-cabbb1910c74/volumes" Jul 17 21:13:47.088683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:47.088642 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:13:47.094665 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:47.094643 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:13:47.095144 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:13:47.095122 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 21:14:23.733817 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.733766 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj"] Jul 17 21:14:23.735680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.734271 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1aa69a-452e-4708-9c30-cabbb1910c74" containerName="console" Jul 17 21:14:23.735680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.734286 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1aa69a-452e-4708-9c30-cabbb1910c74" containerName="console" Jul 17 21:14:23.735680 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.734355 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1aa69a-452e-4708-9c30-cabbb1910c74" containerName="console" Jul 17 21:14:23.736302 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.736285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.738818 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.738793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jfj62\"" Jul 17 21:14:23.738916 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.738796 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 21:14:23.739973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.739957 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:14:23.744838 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.744821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj"] Jul 17 21:14:23.790281 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.790260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkdb\" (UniqueName: \"kubernetes.io/projected/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-kube-api-access-2zkdb\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.790392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.790304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.891275 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.891242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkdb\" (UniqueName: \"kubernetes.io/projected/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-kube-api-access-2zkdb\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.891456 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.891299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.891685 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.891665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:23.899891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:23.899869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkdb\" (UniqueName: \"kubernetes.io/projected/91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9-kube-api-access-2zkdb\") pod \"openshift-lws-operator-fd8ccff4c-tbgsj\" (UID: \"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:24.059283 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:24.059246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" Jul 17 21:14:24.203399 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:24.203370 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj"] Jul 17 21:14:24.206013 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:14:24.205984 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91caccb0_5692_4f1a_ae38_e6ee4b2c7ec9.slice/crio-c56f4fe2c32ba187e88bdac2c6edf2744397311f6a4da4a162b08c79097d5dee WatchSource:0}: Error finding container c56f4fe2c32ba187e88bdac2c6edf2744397311f6a4da4a162b08c79097d5dee: Status 404 returned error can't find the container with id c56f4fe2c32ba187e88bdac2c6edf2744397311f6a4da4a162b08c79097d5dee Jul 17 21:14:24.207512 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:24.207494 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:14:24.786063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:24.786023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" event={"ID":"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9","Type":"ContainerStarted","Data":"c56f4fe2c32ba187e88bdac2c6edf2744397311f6a4da4a162b08c79097d5dee"} Jul 17 21:14:28.805048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:28.805008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" event={"ID":"91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9","Type":"ContainerStarted","Data":"1cde5a52e8e44fff8864487d08380b8daa965c9275a324bc5576d16bdf66f176"} Jul 17 21:14:28.823157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:28.823102 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tbgsj" podStartSLOduration=2.14440727 podStartE2EDuration="5.823088325s" totalImagesPullingTime="3.678681055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:28.82157973 +0000 UTC m=+342.126578652" watchObservedRunningTime="2026-07-17 21:14:28.823088325 +0000 UTC m=+342.128087255" Jul 17 21:14:44.572942 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.572849 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms"] Jul 17 21:14:44.577206 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.577183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.581267 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.581244 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 21:14:44.581409 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.581327 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 21:14:44.583978 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.583899 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 21:14:44.587430 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.587411 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\"" Jul 17 21:14:44.587736 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.587721 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 21:14:44.595746 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.595723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms"] Jul 17 21:14:44.664167 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.664133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.664318 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.664181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.664318 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.664221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.664318 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.664267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.664471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.664342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bj8\" (UniqueName: \"kubernetes.io/projected/3fec7d27-e113-49aa-bf14-2767ee7782cf-kube-api-access-t7bj8\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765412 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765552 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765552 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765552 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bj8\" (UniqueName: \"kubernetes.io/projected/3fec7d27-e113-49aa-bf14-2767ee7782cf-kube-api-access-t7bj8\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765709 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.765932 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.765906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.767910 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.767883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fec7d27-e113-49aa-bf14-2767ee7782cf-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.767991 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.767961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.768147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.768131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fec7d27-e113-49aa-bf14-2767ee7782cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.777152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.777134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bj8\" (UniqueName: \"kubernetes.io/projected/3fec7d27-e113-49aa-bf14-2767ee7782cf-kube-api-access-t7bj8\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-vjsms\" (UID: \"3fec7d27-e113-49aa-bf14-2767ee7782cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:44.887880 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:44.887801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:45.055367 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:45.055347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms"] Jul 17 21:14:45.057161 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:14:45.057140 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fec7d27_e113_49aa_bf14_2767ee7782cf.slice/crio-a5bb924f7c14c885e3c80122e09670fd8570fb4bdff8eb7cb72ad4ffb301bde0 WatchSource:0}: Error finding container a5bb924f7c14c885e3c80122e09670fd8570fb4bdff8eb7cb72ad4ffb301bde0: Status 404 returned error can't find the container with id a5bb924f7c14c885e3c80122e09670fd8570fb4bdff8eb7cb72ad4ffb301bde0 Jul 17 21:14:45.874181 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:45.874131 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" event={"ID":"3fec7d27-e113-49aa-bf14-2767ee7782cf","Type":"ContainerStarted","Data":"a5bb924f7c14c885e3c80122e09670fd8570fb4bdff8eb7cb72ad4ffb301bde0"} Jul 17 21:14:48.890098 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:48.890055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" event={"ID":"3fec7d27-e113-49aa-bf14-2767ee7782cf","Type":"ContainerStarted","Data":"1c85921ee941aa32a6eb1342d8fa4a7d4383eace77ecc1e44f36ad38b0e74863"} Jul 17 21:14:49.896157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:49.896118 2573 generic.go:356] "Generic (PLEG): container finished" podID="3fec7d27-e113-49aa-bf14-2767ee7782cf" containerID="1c85921ee941aa32a6eb1342d8fa4a7d4383eace77ecc1e44f36ad38b0e74863" exitCode=0 Jul 17 21:14:49.896533 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:49.896205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" event={"ID":"3fec7d27-e113-49aa-bf14-2767ee7782cf","Type":"ContainerDied","Data":"1c85921ee941aa32a6eb1342d8fa4a7d4383eace77ecc1e44f36ad38b0e74863"} Jul 17 21:14:50.901996 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:50.901965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" event={"ID":"3fec7d27-e113-49aa-bf14-2767ee7782cf","Type":"ContainerStarted","Data":"4ab333871ee575b638ea7aa971f9e00c147ae4606f268b5ad830a5c6683c9db8"} Jul 17 21:14:50.902363 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:50.902176 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:14:50.928075 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:14:50.928027 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" podStartSLOduration=2.609682198 podStartE2EDuration="6.928012293s" totalImagesPullingTime="3.280493765s" totalInitContainerRuntime="1.03783633s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:50.925159272 +0000 UTC m=+364.230158215" watchObservedRunningTime="2026-07-17 21:14:50.928012293 +0000 UTC m=+364.233011224" Jul 17 21:15:01.909171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:01.909140 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-vjsms" Jul 17 21:15:04.212818 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.212761 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh"] Jul 17 21:15:04.220880 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.220859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.224070 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.224053 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 21:15:04.224981 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.224958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ptj6w\"" Jul 17 21:15:04.224981 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.224977 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 21:15:04.225107 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.224976 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 21:15:04.225107 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.224978 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 21:15:04.227890 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.227869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.228684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.228661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295t2\" (UniqueName: \"kubernetes.io/projected/c57535d6-10b6-4d40-a748-f297653e7d91-kube-api-access-295t2\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.228952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.228929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c57535d6-10b6-4d40-a748-f297653e7d91-tmp\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.229994 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.229971 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh"] Jul 17 21:15:04.329957 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.329907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c57535d6-10b6-4d40-a748-f297653e7d91-tmp\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.330150 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.329989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.330150 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.330010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295t2\" (UniqueName: \"kubernetes.io/projected/c57535d6-10b6-4d40-a748-f297653e7d91-kube-api-access-295t2\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.330150 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:15:04.330130 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 17 21:15:04.330282 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:15:04.330204 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs podName:c57535d6-10b6-4d40-a748-f297653e7d91 nodeName:}" failed. No retries permitted until 2026-07-17 21:15:04.830186268 +0000 UTC m=+378.135185177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs") pod "kube-auth-proxy-7cf476465c-dw7vh" (UID: "c57535d6-10b6-4d40-a748-f297653e7d91") : secret "kube-auth-proxy-tls" not found Jul 17 21:15:04.332239 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.332222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c57535d6-10b6-4d40-a748-f297653e7d91-tmp\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.353367 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.353348 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-295t2\" (UniqueName: \"kubernetes.io/projected/c57535d6-10b6-4d40-a748-f297653e7d91-kube-api-access-295t2\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.834943 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.834909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:04.837412 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:04.837393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57535d6-10b6-4d40-a748-f297653e7d91-tls-certs\") pod \"kube-auth-proxy-7cf476465c-dw7vh\" (UID: \"c57535d6-10b6-4d40-a748-f297653e7d91\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:05.131514 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:05.131430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" Jul 17 21:15:05.278925 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:05.278845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh"] Jul 17 21:15:05.281603 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:15:05.281575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57535d6_10b6_4d40_a748_f297653e7d91.slice/crio-a4d004466bd965ce7930aa52a8ca64e13c537fb79c4fcb1d78786674ce7a3a2b WatchSource:0}: Error finding container a4d004466bd965ce7930aa52a8ca64e13c537fb79c4fcb1d78786674ce7a3a2b: Status 404 returned error can't find the container with id a4d004466bd965ce7930aa52a8ca64e13c537fb79c4fcb1d78786674ce7a3a2b Jul 17 21:15:05.970387 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:05.970352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" event={"ID":"c57535d6-10b6-4d40-a748-f297653e7d91","Type":"ContainerStarted","Data":"a4d004466bd965ce7930aa52a8ca64e13c537fb79c4fcb1d78786674ce7a3a2b"} Jul 17 21:15:08.987232 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:08.987197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" event={"ID":"c57535d6-10b6-4d40-a748-f297653e7d91","Type":"ContainerStarted","Data":"cd18415caccbf32dc1bc13336c696102fe671de2052befb8d84f329a6d5317cc"} Jul 17 21:15:09.002679 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:09.002635 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cf476465c-dw7vh" podStartSLOduration=1.827386087 podStartE2EDuration="5.002617053s" totalImagesPullingTime="3.175230966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:09.001995237 +0000 UTC m=+382.306994170" watchObservedRunningTime="2026-07-17 21:15:09.002617053 +0000 UTC m=+382.307615983" Jul 17 21:15:11.673779 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.673748 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh"] Jul 17 21:15:11.676354 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.676336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.679986 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.679947 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 21:15:11.679986 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.679950 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 21:15:11.680155 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.679985 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\"" Jul 17 21:15:11.680284 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.680270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 21:15:11.690014 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.689988 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh"] Jul 17 21:15:11.796645 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.796614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f08f2cc-22ab-4acf-9181-0d89bf944c97-manager-config\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.796645 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.796645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.796881 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.796679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.796881 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.796733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql9h\" (UniqueName: \"kubernetes.io/projected/1f08f2cc-22ab-4acf-9181-0d89bf944c97-kube-api-access-fql9h\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.897378 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.897349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f08f2cc-22ab-4acf-9181-0d89bf944c97-manager-config\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.897378 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.897381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.897594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.897417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.897594 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.897460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql9h\" (UniqueName: \"kubernetes.io/projected/1f08f2cc-22ab-4acf-9181-0d89bf944c97-kube-api-access-fql9h\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.898140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.898117 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f08f2cc-22ab-4acf-9181-0d89bf944c97-manager-config\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.900074 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.900049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.900172 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.900087 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f08f2cc-22ab-4acf-9181-0d89bf944c97-cert\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.905306 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.905286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql9h\" (UniqueName: \"kubernetes.io/projected/1f08f2cc-22ab-4acf-9181-0d89bf944c97-kube-api-access-fql9h\") pod \"lws-controller-manager-7f487bdccb-btsvh\" (UID: \"1f08f2cc-22ab-4acf-9181-0d89bf944c97\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:11.985792 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:11.985701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:12.119487 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:12.119462 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh"] Jul 17 21:15:12.122214 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:15:12.122174 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f08f2cc_22ab_4acf_9181_0d89bf944c97.slice/crio-25cbe279d56ebc83515d79a24026ad09d393253f4b399a2b1dbec67ecf928607 WatchSource:0}: Error finding container 25cbe279d56ebc83515d79a24026ad09d393253f4b399a2b1dbec67ecf928607: Status 404 returned error can't find the container with id 25cbe279d56ebc83515d79a24026ad09d393253f4b399a2b1dbec67ecf928607 Jul 17 21:15:13.010752 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:13.010711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" event={"ID":"1f08f2cc-22ab-4acf-9181-0d89bf944c97","Type":"ContainerStarted","Data":"25cbe279d56ebc83515d79a24026ad09d393253f4b399a2b1dbec67ecf928607"} Jul 17 21:15:14.016300 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:14.016266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" event={"ID":"1f08f2cc-22ab-4acf-9181-0d89bf944c97","Type":"ContainerStarted","Data":"cbd7a772585267a759311a285b063162177611e49a070a9698f8979d192f3c62"} Jul 17 21:15:14.016866 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:14.016393 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:15:14.048446 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:14.048398 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" podStartSLOduration=1.357269472 podStartE2EDuration="3.048382668s" totalImagesPullingTime="1.691113196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:14.042521848 +0000 UTC m=+387.347520779" watchObservedRunningTime="2026-07-17 21:15:14.048382668 +0000 UTC m=+387.353381599" Jul 17 21:15:25.023285 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:15:25.023255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-btsvh" Jul 17 21:16:15.612314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.612231 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-775d944cb7-hpz2v"] Jul 17 21:16:15.614881 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.614859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.630143 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.630117 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775d944cb7-hpz2v"] Jul 17 21:16:15.767470 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-trusted-ca-bundle\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-console-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-service-ca\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767745 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-oauth-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767745 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-oauth-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.767745 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.767732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmrk\" (UniqueName: \"kubernetes.io/projected/dd57b04c-f02f-4827-8a35-03abbc7198be-kube-api-access-bsmrk\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869281 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-console-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869281 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-service-ca\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869469 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869302 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-oauth-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869469 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-oauth-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869469 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmrk\" (UniqueName: \"kubernetes.io/projected/dd57b04c-f02f-4827-8a35-03abbc7198be-kube-api-access-bsmrk\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869469 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.869469 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.869458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-trusted-ca-bundle\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.870083 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.870050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-oauth-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.870370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.870103 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-console-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.870370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.870119 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-service-ca\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.870370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.870237 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd57b04c-f02f-4827-8a35-03abbc7198be-trusted-ca-bundle\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.871971 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.871949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-serving-cert\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.872076 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.871980 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd57b04c-f02f-4827-8a35-03abbc7198be-console-oauth-config\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.877852 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.877835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmrk\" (UniqueName: \"kubernetes.io/projected/dd57b04c-f02f-4827-8a35-03abbc7198be-kube-api-access-bsmrk\") pod \"console-775d944cb7-hpz2v\" (UID: \"dd57b04c-f02f-4827-8a35-03abbc7198be\") " pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:15.923906 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:15.923873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:16.050617 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:16.050595 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775d944cb7-hpz2v"] Jul 17 21:16:16.052883 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:16:16.052858 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd57b04c_f02f_4827_8a35_03abbc7198be.slice/crio-d006f023039ddcda6b2bcde524e7e4e008f3e270dc0b8c7020f98c7c3dca9ccf WatchSource:0}: Error finding container d006f023039ddcda6b2bcde524e7e4e008f3e270dc0b8c7020f98c7c3dca9ccf: Status 404 returned error can't find the container with id d006f023039ddcda6b2bcde524e7e4e008f3e270dc0b8c7020f98c7c3dca9ccf Jul 17 21:16:16.297895 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:16.297858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775d944cb7-hpz2v" event={"ID":"dd57b04c-f02f-4827-8a35-03abbc7198be","Type":"ContainerStarted","Data":"9a775e26cf1417919fc6be72cbe91e753f99e360ecd04c6dd1e80a21aa673fc7"} Jul 17 21:16:16.298089 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:16.297910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775d944cb7-hpz2v" event={"ID":"dd57b04c-f02f-4827-8a35-03abbc7198be","Type":"ContainerStarted","Data":"d006f023039ddcda6b2bcde524e7e4e008f3e270dc0b8c7020f98c7c3dca9ccf"} Jul 17 21:16:16.317832 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:16.317762 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-775d944cb7-hpz2v" podStartSLOduration=1.317746271 podStartE2EDuration="1.317746271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:16.317101818 +0000 UTC m=+449.622100762" watchObservedRunningTime="2026-07-17 21:16:16.317746271 +0000 UTC m=+449.622745248" Jul 17 21:16:25.924336 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:25.924305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:25.924798 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:25.924387 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:25.928850 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:25.928829 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:26.351587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:26.351559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-775d944cb7-hpz2v" Jul 17 21:16:26.396014 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:26.395982 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:16:51.417499 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.417442 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d4b469b68-hzcjj" podUID="da7d6fff-808d-4af6-9eb1-d868659792f9" containerName="console" containerID="cri-o://968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000" gracePeriod=15 Jul 17 21:16:51.657970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.657947 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:16:51.775397 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775371 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775397 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775433 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775462 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775510 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xgld\" (UniqueName: \"kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775587 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775825 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775597 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config\") pod \"da7d6fff-808d-4af6-9eb1-d868659792f9\" (UID: \"da7d6fff-808d-4af6-9eb1-d868659792f9\") " Jul 17 21:16:51.775894 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775826 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:16:51.775894 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775844 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca" (OuterVolumeSpecName: "service-ca") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:16:51.775894 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.775856 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config" (OuterVolumeSpecName: "console-config") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:16:51.776076 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.776057 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:16:51.777912 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.777871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:16:51.778020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.777936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:16:51.778020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.777940 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld" (OuterVolumeSpecName: "kube-api-access-9xgld") pod "da7d6fff-808d-4af6-9eb1-d868659792f9" (UID: "da7d6fff-808d-4af6-9eb1-d868659792f9"). InnerVolumeSpecName "kube-api-access-9xgld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:16:51.876672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876638 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876664 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876674 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876682 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da7d6fff-808d-4af6-9eb1-d868659792f9-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876693 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgld\" (UniqueName: \"kubernetes.io/projected/da7d6fff-808d-4af6-9eb1-d868659792f9-kube-api-access-9xgld\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876708 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:51.876960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:51.876717 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da7d6fff-808d-4af6-9eb1-d868659792f9-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:16:52.471811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.471750 2573 generic.go:356] "Generic (PLEG): container finished" podID="da7d6fff-808d-4af6-9eb1-d868659792f9" containerID="968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000" exitCode=0 Jul 17 21:16:52.471811 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.471814 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b469b68-hzcjj" Jul 17 21:16:52.472313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.471814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b469b68-hzcjj" event={"ID":"da7d6fff-808d-4af6-9eb1-d868659792f9","Type":"ContainerDied","Data":"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000"} Jul 17 21:16:52.472313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.471866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b469b68-hzcjj" event={"ID":"da7d6fff-808d-4af6-9eb1-d868659792f9","Type":"ContainerDied","Data":"89f0d3c4cce1e8f18d59778155df58d4ad24e940028e4b1ccc5eb9ffe01bb2a7"} Jul 17 21:16:52.472313 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.471887 2573 scope.go:117] "RemoveContainer" containerID="968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000" Jul 17 21:16:52.481473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.481455 2573 scope.go:117] "RemoveContainer" containerID="968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000" Jul 17 21:16:52.481717 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:16:52.481699 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000\": container with ID starting with 968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000 not found: ID does not exist" containerID="968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000" Jul 17 21:16:52.481795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.481732 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000"} err="failed to get container status \"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000\": rpc error: code = NotFound desc = could not find container \"968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000\": container with ID starting with 968feeab705a10e54ca65694b5b3560e910c4a4e67823d63486b5a40ed9e3000 not found: ID does not exist" Jul 17 21:16:52.492653 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.492633 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:16:52.496427 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:52.496407 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4b469b68-hzcjj"] Jul 17 21:16:53.237975 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:16:53.237947 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da7d6fff-808d-4af6-9eb1-d868659792f9" path="/var/lib/kubelet/pods/da7d6fff-808d-4af6-9eb1-d868659792f9/volumes" Jul 17 21:17:03.081977 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.081943 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:03.082371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.082357 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7d6fff-808d-4af6-9eb1-d868659792f9" containerName="console" Jul 17 21:17:03.082417 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.082373 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7d6fff-808d-4af6-9eb1-d868659792f9" containerName="console" Jul 17 21:17:03.082453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.082434 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7d6fff-808d-4af6-9eb1-d868659792f9" containerName="console" Jul 17 21:17:03.085678 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.085645 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.088526 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.088504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tp7kt\"" Jul 17 21:17:03.088674 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.088657 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:17:03.088733 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.088657 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:17:03.102824 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.102796 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:03.180728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.180681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.180927 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.180745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8j9l\" (UniqueName: \"kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.281744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.281707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.281919 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.281793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8j9l\" (UniqueName: \"kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.282125 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.282107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.291233 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.291209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8j9l\" (UniqueName: \"kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tq2pg\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.399058 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.398984 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:03.536998 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:03.536975 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:03.538671 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:17:03.538647 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6829c8b5_baa1_4996_b24c_9976ee491bcf.slice/crio-d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241 WatchSource:0}: Error finding container d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241: Status 404 returned error can't find the container with id d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241 Jul 17 21:17:04.529695 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:04.529656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" event={"ID":"6829c8b5-baa1-4996-b24c-9976ee491bcf","Type":"ContainerStarted","Data":"d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241"} Jul 17 21:17:08.551692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:08.551658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" event={"ID":"6829c8b5-baa1-4996-b24c-9976ee491bcf","Type":"ContainerStarted","Data":"2aa3df9f4a60ca030516f4d3a2961bcaf3d76e87d3f231d4136a0ee5ecf102c6"} Jul 17 21:17:08.552099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:08.551791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:08.573963 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:08.573900 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" podStartSLOduration=1.313879478 podStartE2EDuration="5.573881431s" totalImagesPullingTime="4.260001953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:08.572685677 +0000 UTC m=+501.877684649" watchObservedRunningTime="2026-07-17 21:17:08.573881431 +0000 UTC m=+501.878880363" Jul 17 21:17:19.558799 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:19.558745 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:29.411135 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.411097 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:29.411586 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.411308 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerName="manager" containerID="cri-o://2aa3df9f4a60ca030516f4d3a2961bcaf3d76e87d3f231d4136a0ee5ecf102c6" gracePeriod=10 Jul 17 21:17:29.557060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.557008 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.30:8081/readyz\": dial tcp 10.134.0.30:8081: connect: connection refused" Jul 17 21:17:29.647700 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.647671 2573 generic.go:356] "Generic (PLEG): container finished" podID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerID="2aa3df9f4a60ca030516f4d3a2961bcaf3d76e87d3f231d4136a0ee5ecf102c6" exitCode=0 Jul 17 21:17:29.647870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.647716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" event={"ID":"6829c8b5-baa1-4996-b24c-9976ee491bcf","Type":"ContainerDied","Data":"2aa3df9f4a60ca030516f4d3a2961bcaf3d76e87d3f231d4136a0ee5ecf102c6"} Jul 17 21:17:29.647870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.647785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" event={"ID":"6829c8b5-baa1-4996-b24c-9976ee491bcf","Type":"ContainerDied","Data":"d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241"} Jul 17 21:17:29.647870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.647809 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1d05c53d1768a754b35fd2123cf0ce0df2752af648033c87b11423c9783e241" Jul 17 21:17:29.649860 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.649846 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:29.728766 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.728682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume\") pod \"6829c8b5-baa1-4996-b24c-9976ee491bcf\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " Jul 17 21:17:29.728766 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.728723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8j9l\" (UniqueName: \"kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l\") pod \"6829c8b5-baa1-4996-b24c-9976ee491bcf\" (UID: \"6829c8b5-baa1-4996-b24c-9976ee491bcf\") " Jul 17 21:17:29.729144 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.729120 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6829c8b5-baa1-4996-b24c-9976ee491bcf" (UID: "6829c8b5-baa1-4996-b24c-9976ee491bcf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:17:29.731011 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.730989 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l" (OuterVolumeSpecName: "kube-api-access-f8j9l") pod "6829c8b5-baa1-4996-b24c-9976ee491bcf" (UID: "6829c8b5-baa1-4996-b24c-9976ee491bcf"). InnerVolumeSpecName "kube-api-access-f8j9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:17:29.829625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.829588 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6829c8b5-baa1-4996-b24c-9976ee491bcf-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:17:29.829625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:29.829620 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8j9l\" (UniqueName: \"kubernetes.io/projected/6829c8b5-baa1-4996-b24c-9976ee491bcf-kube-api-access-f8j9l\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:17:30.652995 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:30.652970 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg" Jul 17 21:17:30.695024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:30.695001 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:30.703939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:30.703915 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tq2pg"] Jul 17 21:17:31.237845 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:31.237809 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" path="/var/lib/kubelet/pods/6829c8b5-baa1-4996-b24c-9976ee491bcf/volumes" Jul 17 21:17:45.741463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.741388 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf"] Jul 17 21:17:45.741838 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.741746 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerName="manager" Jul 17 21:17:45.741838 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.741758 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerName="manager" Jul 17 21:17:45.741918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.741868 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6829c8b5-baa1-4996-b24c-9976ee491bcf" containerName="manager" Jul 17 21:17:45.743909 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.743890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.746959 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.746935 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ldvks\"" Jul 17 21:17:45.747264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.747247 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 21:17:45.756824 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.756804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf"] Jul 17 21:17:45.868987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.868954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.868987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.868990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6vq\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-kube-api-access-tp6vq\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.869362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.869273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6vq\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-kube-api-access-tp6vq\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.969954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.969948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970260 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970260 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970127 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970260 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970413 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970362 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970413 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.970761 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.970740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.972500 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.972478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.972717 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.972695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.978193 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.978173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:45.978685 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:45.978662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6vq\" (UniqueName: \"kubernetes.io/projected/90ed55e8-e67e-407b-bf8b-e2fc121f47ca-kube-api-access-tp6vq\") pod \"maas-default-gateway-openshift-default-56654dcf7-8grmf\" (UID: \"90ed55e8-e67e-407b-bf8b-e2fc121f47ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:46.053618 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:46.053592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:46.198367 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:46.198343 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf"] Jul 17 21:17:46.726360 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:46.726324 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" event={"ID":"90ed55e8-e67e-407b-bf8b-e2fc121f47ca","Type":"ContainerStarted","Data":"ac03545d601284beacfba93b7ab25639d374bdc4b97734216f608931d240ef13"} Jul 17 21:17:53.843264 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:53.843220 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:17:53.843550 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:53.843304 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:17:53.843550 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:53.843338 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:17:54.768312 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:54.768270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" event={"ID":"90ed55e8-e67e-407b-bf8b-e2fc121f47ca","Type":"ContainerStarted","Data":"ee3c60e24cb660765bdde64085c2092f7a6e75ed46165f99ab58a511ca6b4543"} Jul 17 21:17:54.789298 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:54.789237 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" podStartSLOduration=2.152510883 podStartE2EDuration="9.789219086s" totalImagesPullingTime="7.636708203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:54.789031157 +0000 UTC m=+548.094030092" watchObservedRunningTime="2026-07-17 21:17:54.789219086 +0000 UTC m=+548.094218091" Jul 17 21:17:55.053833 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.053732 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:55.058349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.058324 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:55.768795 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.768744 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:17:55.771427 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.771405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:55.773197 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.773160 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:55.773977 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.773961 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:17:55.774337 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.774317 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:17:55.774959 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.774944 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h285x\"" Jul 17 21:17:55.775054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.774949 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:17:55.775054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.774949 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 21:17:55.781958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.781938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:17:55.861025 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.860984 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:17:55.967391 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.967342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:55.967564 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:55.967411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkdl\" (UniqueName: \"kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.068369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.068239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.068369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.068364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkdl\" (UniqueName: \"kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.068853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.068835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.076106 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.076082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkdl\" (UniqueName: \"kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl\") pod \"limitador-limitador-64c94fd64f-pj2ff\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.081937 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.081913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:17:56.211018 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.210992 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:17:56.212919 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:17:56.212887 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf807a8b_5b14_4cd6_9286_f1711e4820dc.slice/crio-42630c9a07a1c37aa57aaaf9bdc48e4773b61cef19fe53a6bd31942d76ca1ec4 WatchSource:0}: Error finding container 42630c9a07a1c37aa57aaaf9bdc48e4773b61cef19fe53a6bd31942d76ca1ec4: Status 404 returned error can't find the container with id 42630c9a07a1c37aa57aaaf9bdc48e4773b61cef19fe53a6bd31942d76ca1ec4 Jul 17 21:17:56.781864 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.781802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" event={"ID":"af807a8b-5b14-4cd6-9286-f1711e4820dc","Type":"ContainerStarted","Data":"42630c9a07a1c37aa57aaaf9bdc48e4773b61cef19fe53a6bd31942d76ca1ec4"} Jul 17 21:17:56.787809 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.787762 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:17:56.790808 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.790787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:17:56.793840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.793722 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nrgzg\"" Jul 17 21:17:56.798617 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.798587 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:17:56.876678 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.876642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fn5\" (UniqueName: \"kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5\") pod \"authorino-75d556bd77-frtnw\" (UID: \"42135f2a-aa05-4149-851a-de0796fe206d\") " pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:17:56.977596 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.977527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fn5\" (UniqueName: \"kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5\") pod \"authorino-75d556bd77-frtnw\" (UID: \"42135f2a-aa05-4149-851a-de0796fe206d\") " pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:17:56.987497 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:56.987465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fn5\" (UniqueName: \"kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5\") pod \"authorino-75d556bd77-frtnw\" (UID: \"42135f2a-aa05-4149-851a-de0796fe206d\") " pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:17:57.110026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:57.109946 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:17:57.266720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:57.266694 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:17:57.269202 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:17:57.269173 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42135f2a_aa05_4149_851a_de0796fe206d.slice/crio-a47561e113fe3d84e2f6f844b24128b5208d6dfd4e74d1a392cb76e2bb481e6b WatchSource:0}: Error finding container a47561e113fe3d84e2f6f844b24128b5208d6dfd4e74d1a392cb76e2bb481e6b: Status 404 returned error can't find the container with id a47561e113fe3d84e2f6f844b24128b5208d6dfd4e74d1a392cb76e2bb481e6b Jul 17 21:17:57.791859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:57.791818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-frtnw" event={"ID":"42135f2a-aa05-4149-851a-de0796fe206d","Type":"ContainerStarted","Data":"a47561e113fe3d84e2f6f844b24128b5208d6dfd4e74d1a392cb76e2bb481e6b"} Jul 17 21:17:59.807489 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:59.807447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" event={"ID":"af807a8b-5b14-4cd6-9286-f1711e4820dc","Type":"ContainerStarted","Data":"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134"} Jul 17 21:17:59.808197 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:17:59.807625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:18:01.819026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:01.818983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-frtnw" event={"ID":"42135f2a-aa05-4149-851a-de0796fe206d","Type":"ContainerStarted","Data":"031e27e143f3b444f6827c20fc5eafb3d3f70fab216048cbc0ea02af0964086a"} Jul 17 21:18:01.836266 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:01.836213 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-frtnw" podStartSLOduration=2.289819535 podStartE2EDuration="5.836195971s" totalImagesPullingTime="3.546376436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:01.835290815 +0000 UTC m=+555.140289747" watchObservedRunningTime="2026-07-17 21:18:01.836195971 +0000 UTC m=+555.141194902" Jul 17 21:18:01.837291 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:01.837256 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" podStartSLOduration=4.216053639 podStartE2EDuration="6.837243451s" totalImagesPullingTime="2.621189812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:59.824581915 +0000 UTC m=+553.129580847" watchObservedRunningTime="2026-07-17 21:18:01.837243451 +0000 UTC m=+555.142242382" Jul 17 21:18:10.671892 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:10.671859 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:18:10.672368 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:10.672131 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" podUID="af807a8b-5b14-4cd6-9286-f1711e4820dc" containerName="limitador" containerID="cri-o://1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134" gracePeriod=30 Jul 17 21:18:10.672908 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:10.672793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:18:11.232421 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.232366 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:18:11.299597 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.299568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file\") pod \"af807a8b-5b14-4cd6-9286-f1711e4820dc\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " Jul 17 21:18:11.299748 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.299607 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wkdl\" (UniqueName: \"kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl\") pod \"af807a8b-5b14-4cd6-9286-f1711e4820dc\" (UID: \"af807a8b-5b14-4cd6-9286-f1711e4820dc\") " Jul 17 21:18:11.300003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.299977 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file" (OuterVolumeSpecName: "config-file") pod "af807a8b-5b14-4cd6-9286-f1711e4820dc" (UID: "af807a8b-5b14-4cd6-9286-f1711e4820dc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:18:11.301786 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.301753 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl" (OuterVolumeSpecName: "kube-api-access-6wkdl") pod "af807a8b-5b14-4cd6-9286-f1711e4820dc" (UID: "af807a8b-5b14-4cd6-9286-f1711e4820dc"). InnerVolumeSpecName "kube-api-access-6wkdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:18:11.401020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.400984 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkdl\" (UniqueName: \"kubernetes.io/projected/af807a8b-5b14-4cd6-9286-f1711e4820dc-kube-api-access-6wkdl\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:18:11.401020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.401014 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af807a8b-5b14-4cd6-9286-f1711e4820dc-config-file\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:18:11.867067 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.867038 2573 generic.go:356] "Generic (PLEG): container finished" podID="af807a8b-5b14-4cd6-9286-f1711e4820dc" containerID="1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134" exitCode=0 Jul 17 21:18:11.867468 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.867081 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" Jul 17 21:18:11.867468 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.867103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" event={"ID":"af807a8b-5b14-4cd6-9286-f1711e4820dc","Type":"ContainerDied","Data":"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134"} Jul 17 21:18:11.867468 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.867148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pj2ff" event={"ID":"af807a8b-5b14-4cd6-9286-f1711e4820dc","Type":"ContainerDied","Data":"42630c9a07a1c37aa57aaaf9bdc48e4773b61cef19fe53a6bd31942d76ca1ec4"} Jul 17 21:18:11.867468 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.867174 2573 scope.go:117] "RemoveContainer" containerID="1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134" Jul 17 21:18:11.876537 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.876519 2573 scope.go:117] "RemoveContainer" containerID="1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134" Jul 17 21:18:11.876763 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:18:11.876747 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134\": container with ID starting with 1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134 not found: ID does not exist" containerID="1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134" Jul 17 21:18:11.876871 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.876790 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134"} err="failed to get container status \"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134\": rpc error: code = NotFound desc = could not find container \"1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134\": container with ID starting with 1c365eab552c2b96f1c2241879216c818336dac7678e086092077dc7c091b134 not found: ID does not exist" Jul 17 21:18:11.889995 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.889969 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:18:11.893879 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:11.893861 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pj2ff"] Jul 17 21:18:12.288108 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.288077 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl"] Jul 17 21:18:12.288500 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.288486 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af807a8b-5b14-4cd6-9286-f1711e4820dc" containerName="limitador" Jul 17 21:18:12.288544 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.288504 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af807a8b-5b14-4cd6-9286-f1711e4820dc" containerName="limitador" Jul 17 21:18:12.288589 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.288580 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af807a8b-5b14-4cd6-9286-f1711e4820dc" containerName="limitador" Jul 17 21:18:12.291501 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.291484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.294315 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.294292 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 21:18:12.294459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.294351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kvd2g\"" Jul 17 21:18:12.294459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.294357 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 21:18:12.294459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.294393 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 17 21:18:12.297828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.297803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl"] Jul 17 21:18:12.411010 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.410975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzpt\" (UniqueName: \"kubernetes.io/projected/6b5f708a-4f3a-48af-8c29-a69db4d71e63-kube-api-access-fhzpt\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.411153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.411050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b5f708a-4f3a-48af-8c29-a69db4d71e63-data\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.512133 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.512102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b5f708a-4f3a-48af-8c29-a69db4d71e63-data\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.512304 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.512166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzpt\" (UniqueName: \"kubernetes.io/projected/6b5f708a-4f3a-48af-8c29-a69db4d71e63-kube-api-access-fhzpt\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.512540 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.512522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b5f708a-4f3a-48af-8c29-a69db4d71e63-data\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.520614 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.520585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzpt\" (UniqueName: \"kubernetes.io/projected/6b5f708a-4f3a-48af-8c29-a69db4d71e63-kube-api-access-fhzpt\") pod \"postgres-69fb4594cf-kzkgl\" (UID: \"6b5f708a-4f3a-48af-8c29-a69db4d71e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.603250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.603181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:12.730031 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.730007 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl"] Jul 17 21:18:12.731954 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:18:12.731927 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5f708a_4f3a_48af_8c29_a69db4d71e63.slice/crio-7037efa6175b4da846f1160faf434e2cde184795a0b19baf96de1a09f3e4ce71 WatchSource:0}: Error finding container 7037efa6175b4da846f1160faf434e2cde184795a0b19baf96de1a09f3e4ce71: Status 404 returned error can't find the container with id 7037efa6175b4da846f1160faf434e2cde184795a0b19baf96de1a09f3e4ce71 Jul 17 21:18:12.874211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:12.874131 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" event={"ID":"6b5f708a-4f3a-48af-8c29-a69db4d71e63","Type":"ContainerStarted","Data":"7037efa6175b4da846f1160faf434e2cde184795a0b19baf96de1a09f3e4ce71"} Jul 17 21:18:13.238307 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:13.238225 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af807a8b-5b14-4cd6-9286-f1711e4820dc" path="/var/lib/kubelet/pods/af807a8b-5b14-4cd6-9286-f1711e4820dc/volumes" Jul 17 21:18:19.917458 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:19.917365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" event={"ID":"6b5f708a-4f3a-48af-8c29-a69db4d71e63","Type":"ContainerStarted","Data":"bc98fc49f1da5d65cc1f41d8136a169a1224ea903017da4d3c5032f9ece9c933"} Jul 17 21:18:19.917458 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:19.917441 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:19.935760 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:19.935716 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" podStartSLOduration=1.090781908 podStartE2EDuration="7.935700869s" totalImagesPullingTime="6.844918961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:19.933930966 +0000 UTC m=+573.238929902" watchObservedRunningTime="2026-07-17 21:18:19.935700869 +0000 UTC m=+573.240699831" Jul 17 21:18:25.953693 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:25.953658 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kzkgl" Jul 17 21:18:28.800583 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:28.800547 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:18:28.801096 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:28.800719 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-frtnw" podUID="42135f2a-aa05-4149-851a-de0796fe206d" containerName="authorino" containerID="cri-o://031e27e143f3b444f6827c20fc5eafb3d3f70fab216048cbc0ea02af0964086a" gracePeriod=30 Jul 17 21:18:28.962742 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:28.962711 2573 generic.go:356] "Generic (PLEG): container finished" podID="42135f2a-aa05-4149-851a-de0796fe206d" containerID="031e27e143f3b444f6827c20fc5eafb3d3f70fab216048cbc0ea02af0964086a" exitCode=0 Jul 17 21:18:28.962911 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:28.962753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-frtnw" event={"ID":"42135f2a-aa05-4149-851a-de0796fe206d","Type":"ContainerDied","Data":"031e27e143f3b444f6827c20fc5eafb3d3f70fab216048cbc0ea02af0964086a"} Jul 17 21:18:29.042302 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.042275 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:18:29.174131 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.174052 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8fn5\" (UniqueName: \"kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5\") pod \"42135f2a-aa05-4149-851a-de0796fe206d\" (UID: \"42135f2a-aa05-4149-851a-de0796fe206d\") " Jul 17 21:18:29.176238 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.176214 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5" (OuterVolumeSpecName: "kube-api-access-j8fn5") pod "42135f2a-aa05-4149-851a-de0796fe206d" (UID: "42135f2a-aa05-4149-851a-de0796fe206d"). InnerVolumeSpecName "kube-api-access-j8fn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:18:29.275008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.274978 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8fn5\" (UniqueName: \"kubernetes.io/projected/42135f2a-aa05-4149-851a-de0796fe206d-kube-api-access-j8fn5\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:18:29.969006 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.968979 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-frtnw" Jul 17 21:18:29.969459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.969020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-frtnw" event={"ID":"42135f2a-aa05-4149-851a-de0796fe206d","Type":"ContainerDied","Data":"a47561e113fe3d84e2f6f844b24128b5208d6dfd4e74d1a392cb76e2bb481e6b"} Jul 17 21:18:29.969459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.969074 2573 scope.go:117] "RemoveContainer" containerID="031e27e143f3b444f6827c20fc5eafb3d3f70fab216048cbc0ea02af0964086a" Jul 17 21:18:29.987362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.987339 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:18:29.990995 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:29.990975 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-frtnw"] Jul 17 21:18:31.238259 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:31.238225 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42135f2a-aa05-4149-851a-de0796fe206d" path="/var/lib/kubelet/pods/42135f2a-aa05-4149-851a-de0796fe206d/volumes" Jul 17 21:18:33.349441 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.349408 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:33.349829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.349814 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42135f2a-aa05-4149-851a-de0796fe206d" containerName="authorino" Jul 17 21:18:33.349829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.349831 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42135f2a-aa05-4149-851a-de0796fe206d" containerName="authorino" Jul 17 21:18:33.349920 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.349911 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42135f2a-aa05-4149-851a-de0796fe206d" containerName="authorino" Jul 17 21:18:33.430280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.430245 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:33.430442 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.430359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.434750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.434727 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:18:33.434918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.434803 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ntt\"" Jul 17 21:18:33.434918 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.434809 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:18:33.537827 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.537797 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:18:33.559744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.559716 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:18:33.559913 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.559828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.616844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.616760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.616844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.616837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9bx\" (UniqueName: \"kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.717727 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.717694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72lc\" (UniqueName: \"kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.717883 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.717735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.717883 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.717816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.717883 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.717856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9bx\" (UniqueName: \"kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.720205 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.720178 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.726364 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.726345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9bx\" (UniqueName: \"kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx\") pod \"maas-controller-d87d8b8fb-bx5gd\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.741329 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.741312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:33.818971 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.818933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.819136 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.819112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72lc\" (UniqueName: \"kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.822057 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.822015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.830744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.830712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72lc\" (UniqueName: \"kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc\") pod \"maas-controller-557f54c68b-p2rvk\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.870040 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.869977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:33.886091 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.886059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:33.887669 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:18:33.887648 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f70d0e_6225_459b_b52c_621e91a1f8a6.slice/crio-bdcc00d2a5e04d0d530bf8fdb1a77c30ac59cc79444da50c08cda4d5eaa42180 WatchSource:0}: Error finding container bdcc00d2a5e04d0d530bf8fdb1a77c30ac59cc79444da50c08cda4d5eaa42180: Status 404 returned error can't find the container with id bdcc00d2a5e04d0d530bf8fdb1a77c30ac59cc79444da50c08cda4d5eaa42180 Jul 17 21:18:33.991543 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:33.991510 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" event={"ID":"25f70d0e-6225-459b-b52c-621e91a1f8a6","Type":"ContainerStarted","Data":"bdcc00d2a5e04d0d530bf8fdb1a77c30ac59cc79444da50c08cda4d5eaa42180"} Jul 17 21:18:34.023045 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:34.023021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:18:34.025385 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:18:34.025357 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01b9964_da50_4279_be4f_e4c9a696cbb8.slice/crio-587e3ceee21b1679ef2f39b31cbf287961cf1bf5fcd2c6f22d06ed617b972d0a WatchSource:0}: Error finding container 587e3ceee21b1679ef2f39b31cbf287961cf1bf5fcd2c6f22d06ed617b972d0a: Status 404 returned error can't find the container with id 587e3ceee21b1679ef2f39b31cbf287961cf1bf5fcd2c6f22d06ed617b972d0a Jul 17 21:18:35.000065 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:35.000024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-p2rvk" event={"ID":"b01b9964-da50-4279-be4f-e4c9a696cbb8","Type":"ContainerStarted","Data":"587e3ceee21b1679ef2f39b31cbf287961cf1bf5fcd2c6f22d06ed617b972d0a"} Jul 17 21:18:37.011724 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.011686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-p2rvk" event={"ID":"b01b9964-da50-4279-be4f-e4c9a696cbb8","Type":"ContainerStarted","Data":"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38"} Jul 17 21:18:37.012203 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.011818 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:37.013370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.013346 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" event={"ID":"25f70d0e-6225-459b-b52c-621e91a1f8a6","Type":"ContainerStarted","Data":"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed"} Jul 17 21:18:37.013502 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.013425 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:37.033137 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.033097 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-557f54c68b-p2rvk" podStartSLOduration=1.537018738 podStartE2EDuration="4.033084235s" totalImagesPullingTime="2.496065497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:37.03166921 +0000 UTC m=+590.336668155" watchObservedRunningTime="2026-07-17 21:18:37.033084235 +0000 UTC m=+590.338083165" Jul 17 21:18:37.051629 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:37.051564 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" podStartSLOduration=1.422630522 podStartE2EDuration="4.051550164s" totalImagesPullingTime="2.628919642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:37.050745151 +0000 UTC m=+590.355744083" watchObservedRunningTime="2026-07-17 21:18:37.051550164 +0000 UTC m=+590.356549095" Jul 17 21:18:47.158554 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:47.158522 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:18:47.159011 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:47.158989 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:18:48.020793 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.020752 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:48.020958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.020837 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:18:48.164020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.163993 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:48.164418 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.164206 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" podUID="25f70d0e-6225-459b-b52c-621e91a1f8a6" containerName="manager" containerID="cri-o://d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed" gracePeriod=30 Jul 17 21:18:48.412168 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.412145 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:48.450457 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.450428 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert\") pod \"25f70d0e-6225-459b-b52c-621e91a1f8a6\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " Jul 17 21:18:48.450607 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.450527 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp9bx\" (UniqueName: \"kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx\") pod \"25f70d0e-6225-459b-b52c-621e91a1f8a6\" (UID: \"25f70d0e-6225-459b-b52c-621e91a1f8a6\") " Jul 17 21:18:48.452698 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.452665 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "25f70d0e-6225-459b-b52c-621e91a1f8a6" (UID: "25f70d0e-6225-459b-b52c-621e91a1f8a6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:18:48.452819 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.452712 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx" (OuterVolumeSpecName: "kube-api-access-mp9bx") pod "25f70d0e-6225-459b-b52c-621e91a1f8a6" (UID: "25f70d0e-6225-459b-b52c-621e91a1f8a6"). InnerVolumeSpecName "kube-api-access-mp9bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:18:48.552054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.551969 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9bx\" (UniqueName: \"kubernetes.io/projected/25f70d0e-6225-459b-b52c-621e91a1f8a6-kube-api-access-mp9bx\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:18:48.552054 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:48.551999 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25f70d0e-6225-459b-b52c-621e91a1f8a6-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:18:49.074032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.073999 2573 generic.go:356] "Generic (PLEG): container finished" podID="25f70d0e-6225-459b-b52c-621e91a1f8a6" containerID="d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed" exitCode=0 Jul 17 21:18:49.074184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.074060 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" Jul 17 21:18:49.074184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.074089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" event={"ID":"25f70d0e-6225-459b-b52c-621e91a1f8a6","Type":"ContainerDied","Data":"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed"} Jul 17 21:18:49.074184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.074169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87d8b8fb-bx5gd" event={"ID":"25f70d0e-6225-459b-b52c-621e91a1f8a6","Type":"ContainerDied","Data":"bdcc00d2a5e04d0d530bf8fdb1a77c30ac59cc79444da50c08cda4d5eaa42180"} Jul 17 21:18:49.074302 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.074192 2573 scope.go:117] "RemoveContainer" containerID="d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed" Jul 17 21:18:49.083395 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.083378 2573 scope.go:117] "RemoveContainer" containerID="d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed" Jul 17 21:18:49.083683 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:18:49.083664 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed\": container with ID starting with d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed not found: ID does not exist" containerID="d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed" Jul 17 21:18:49.083728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.083691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed"} err="failed to get container status \"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed\": rpc error: code = NotFound desc = could not find container \"d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed\": container with ID starting with d3e4f5707feffb7028c4cce0780b9f69dab75efd7ceca71381249864b5048bed not found: ID does not exist" Jul 17 21:18:49.097785 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.097748 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:49.101382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.101361 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d87d8b8fb-bx5gd"] Jul 17 21:18:49.237697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:18:49.237665 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25f70d0e-6225-459b-b52c-621e91a1f8a6" path="/var/lib/kubelet/pods/25f70d0e-6225-459b-b52c-621e91a1f8a6/volumes" Jul 17 21:19:09.381489 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.381408 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk"] Jul 17 21:19:09.381977 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.381792 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f70d0e-6225-459b-b52c-621e91a1f8a6" containerName="manager" Jul 17 21:19:09.381977 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.381807 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f70d0e-6225-459b-b52c-621e91a1f8a6" containerName="manager" Jul 17 21:19:09.381977 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.381883 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f70d0e-6225-459b-b52c-621e91a1f8a6" containerName="manager" Jul 17 21:19:09.384100 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.384084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.395243 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.395225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mbgfd\"" Jul 17 21:19:09.396371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.396358 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 21:19:09.401896 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.401876 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk"] Jul 17 21:19:09.543620 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.543588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20481527-9c54-4ccd-946e-4b79c0040d2f-openshift-service-ca\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.543620 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.543624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp6f\" (UniqueName: \"kubernetes.io/projected/20481527-9c54-4ccd-946e-4b79c0040d2f-kube-api-access-scp6f\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.543840 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.543666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20481527-9c54-4ccd-946e-4b79c0040d2f-maas-api-tls\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.644440 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.644358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20481527-9c54-4ccd-946e-4b79c0040d2f-maas-api-tls\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.644582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.644442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20481527-9c54-4ccd-946e-4b79c0040d2f-openshift-service-ca\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.644582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.644462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp6f\" (UniqueName: \"kubernetes.io/projected/20481527-9c54-4ccd-946e-4b79c0040d2f-kube-api-access-scp6f\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.645159 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.645140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20481527-9c54-4ccd-946e-4b79c0040d2f-openshift-service-ca\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.646926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.646906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20481527-9c54-4ccd-946e-4b79c0040d2f-maas-api-tls\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.652543 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.652521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp6f\" (UniqueName: \"kubernetes.io/projected/20481527-9c54-4ccd-946e-4b79c0040d2f-kube-api-access-scp6f\") pod \"maas-api-6f68cd45b9-4hwjk\" (UID: \"20481527-9c54-4ccd-946e-4b79c0040d2f\") " pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.693970 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.693945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:09.829324 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.829298 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk"] Jul 17 21:19:09.831496 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:09.831470 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20481527_9c54_4ccd_946e_4b79c0040d2f.slice/crio-0716d50e8d7be4bd557cb5fca775a6d281f0dda73321efc83b040bb1b9ce7afd WatchSource:0}: Error finding container 0716d50e8d7be4bd557cb5fca775a6d281f0dda73321efc83b040bb1b9ce7afd: Status 404 returned error can't find the container with id 0716d50e8d7be4bd557cb5fca775a6d281f0dda73321efc83b040bb1b9ce7afd Jul 17 21:19:09.859084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.859058 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-86qlf"] Jul 17 21:19:09.862225 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.862209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:09.865079 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.865058 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 21:19:09.871651 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.871629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-86qlf"] Jul 17 21:19:09.874392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.874370 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-72zq9"] Jul 17 21:19:09.877055 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.877040 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:09.879623 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.879606 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tz99s\"" Jul 17 21:19:09.886927 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.886843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-72zq9"] Jul 17 21:19:09.947026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.946944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskms\" (UniqueName: \"kubernetes.io/projected/1fafc772-0bfc-433d-9462-e5922835c535-kube-api-access-xskms\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:09.947026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:09.946980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fafc772-0bfc-433d-9462-e5922835c535-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.047986 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.047953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fafc772-0bfc-433d-9462-e5922835c535-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.048147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.048000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prsx\" (UniqueName: \"kubernetes.io/projected/14411f37-d3fb-4c59-be8d-6662499ca5a9-kube-api-access-7prsx\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.048147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.048114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14411f37-d3fb-4c59-be8d-6662499ca5a9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.048232 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.048192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskms\" (UniqueName: \"kubernetes.io/projected/1fafc772-0bfc-433d-9462-e5922835c535-kube-api-access-xskms\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.048608 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.048585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fafc772-0bfc-433d-9462-e5922835c535-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.057198 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.057176 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskms\" (UniqueName: \"kubernetes.io/projected/1fafc772-0bfc-433d-9462-e5922835c535-kube-api-access-xskms\") pod \"payload-pre-processing-7677b774bb-86qlf\" (UID: \"1fafc772-0bfc-433d-9462-e5922835c535\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.149274 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.149240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prsx\" (UniqueName: \"kubernetes.io/projected/14411f37-d3fb-4c59-be8d-6662499ca5a9-kube-api-access-7prsx\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.149274 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.149277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14411f37-d3fb-4c59-be8d-6662499ca5a9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.149944 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.149926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14411f37-d3fb-4c59-be8d-6662499ca5a9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.159626 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.159603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prsx\" (UniqueName: \"kubernetes.io/projected/14411f37-d3fb-4c59-be8d-6662499ca5a9-kube-api-access-7prsx\") pod \"payload-processing-5bf47cd988-72zq9\" (UID: \"14411f37-d3fb-4c59-be8d-6662499ca5a9\") " pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.174217 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.174183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:10.175360 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.175332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" event={"ID":"20481527-9c54-4ccd-946e-4b79c0040d2f","Type":"ContainerStarted","Data":"0716d50e8d7be4bd557cb5fca775a6d281f0dda73321efc83b040bb1b9ce7afd"} Jul 17 21:19:10.188878 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.188857 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:10.312465 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.312437 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-86qlf"] Jul 17 21:19:10.315845 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:10.315814 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fafc772_0bfc_433d_9462_e5922835c535.slice/crio-806969f4f452dfa96ff77a36687296526593722e69b033db4e6e9b813835534d WatchSource:0}: Error finding container 806969f4f452dfa96ff77a36687296526593722e69b033db4e6e9b813835534d: Status 404 returned error can't find the container with id 806969f4f452dfa96ff77a36687296526593722e69b033db4e6e9b813835534d Jul 17 21:19:10.562758 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:10.562608 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-72zq9"] Jul 17 21:19:10.585267 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:10.585170 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14411f37_d3fb_4c59_be8d_6662499ca5a9.slice/crio-3ab3bd3e52f1c9a306dd08dcf4f5a040188c12a594dc9cdf5e1becd049c39cdf WatchSource:0}: Error finding container 3ab3bd3e52f1c9a306dd08dcf4f5a040188c12a594dc9cdf5e1becd049c39cdf: Status 404 returned error can't find the container with id 3ab3bd3e52f1c9a306dd08dcf4f5a040188c12a594dc9cdf5e1becd049c39cdf Jul 17 21:19:11.189870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:11.189764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" event={"ID":"14411f37-d3fb-4c59-be8d-6662499ca5a9","Type":"ContainerStarted","Data":"3ab3bd3e52f1c9a306dd08dcf4f5a040188c12a594dc9cdf5e1becd049c39cdf"} Jul 17 21:19:11.191658 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:11.191596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" event={"ID":"1fafc772-0bfc-433d-9462-e5922835c535","Type":"ContainerStarted","Data":"806969f4f452dfa96ff77a36687296526593722e69b033db4e6e9b813835534d"} Jul 17 21:19:14.210562 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.210524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" event={"ID":"1fafc772-0bfc-433d-9462-e5922835c535","Type":"ContainerStarted","Data":"b2f9b6b2dbc3ec5482fa19d86e6f6b9131ca81fd2bb095853754f6068e8d3a7a"} Jul 17 21:19:14.211048 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.210587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:14.212166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.212143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" event={"ID":"14411f37-d3fb-4c59-be8d-6662499ca5a9","Type":"ContainerStarted","Data":"eda2bf87dd6add8eb722bc85812d42996a37ba1260491e4e1676b1484e9368a4"} Jul 17 21:19:14.212295 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.212276 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:14.213678 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.213656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" event={"ID":"20481527-9c54-4ccd-946e-4b79c0040d2f","Type":"ContainerStarted","Data":"43aa7f680008f3deebd54e7b34fdc301375c6721a5c03565d5402137eafa74e6"} Jul 17 21:19:14.213809 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.213795 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:14.228786 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.228731 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" podStartSLOduration=1.853148923 podStartE2EDuration="5.22871777s" totalImagesPullingTime="3.375568847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:14.226830316 +0000 UTC m=+627.531829242" watchObservedRunningTime="2026-07-17 21:19:14.22871777 +0000 UTC m=+627.533716699" Jul 17 21:19:14.243950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.243912 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" podStartSLOduration=1.44958992 podStartE2EDuration="5.243904122s" totalImagesPullingTime="3.794314202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:14.243520524 +0000 UTC m=+627.548519456" watchObservedRunningTime="2026-07-17 21:19:14.243904122 +0000 UTC m=+627.548903052" Jul 17 21:19:14.260743 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:14.260698 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" podStartSLOduration=2.145286956 podStartE2EDuration="5.260681483s" totalImagesPullingTime="3.115394527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:14.260194315 +0000 UTC m=+627.565193265" watchObservedRunningTime="2026-07-17 21:19:14.260681483 +0000 UTC m=+627.565680407" Jul 17 21:19:18.235959 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.235925 2573 generic.go:356] "Generic (PLEG): container finished" podID="90ed55e8-e67e-407b-bf8b-e2fc121f47ca" containerID="ee3c60e24cb660765bdde64085c2092f7a6e75ed46165f99ab58a511ca6b4543" exitCode=0 Jul 17 21:19:18.236494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.235967 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" event={"ID":"90ed55e8-e67e-407b-bf8b-e2fc121f47ca","Type":"ContainerDied","Data":"ee3c60e24cb660765bdde64085c2092f7a6e75ed46165f99ab58a511ca6b4543"} Jul 17 21:19:18.236574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.236496 2573 scope.go:117] "RemoveContainer" containerID="ee3c60e24cb660765bdde64085c2092f7a6e75ed46165f99ab58a511ca6b4543" Jul 17 21:19:18.237502 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.237475 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:19:18.237566 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.237549 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:19:18.237603 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:18.237578 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:19:19.241844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:19.241813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" event={"ID":"90ed55e8-e67e-407b-bf8b-e2fc121f47ca","Type":"ContainerStarted","Data":"79429961fed70cb9ff19fa848e09a7c92c0eaaa62d1d6ebe265bfbf4e57e0fc6"} Jul 17 21:19:20.054838 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:20.054794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:19:20.055072 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:20.055043 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8grmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 21:19:20.055149 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:20.055122 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" podUID="90ed55e8-e67e-407b-bf8b-e2fc121f47ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 21:19:20.225433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:20.225406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f68cd45b9-4hwjk" Jul 17 21:19:21.054761 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:21.054715 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8grmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 21:19:21.055303 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:21.054820 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" podUID="90ed55e8-e67e-407b-bf8b-e2fc121f47ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 21:19:22.054921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:22.054876 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8grmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 21:19:22.055405 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:22.054947 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" podUID="90ed55e8-e67e-407b-bf8b-e2fc121f47ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 21:19:23.055152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:23.054653 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8grmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 21:19:23.055152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:23.054718 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" podUID="90ed55e8-e67e-407b-bf8b-e2fc121f47ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 21:19:24.058538 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:24.058509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:19:24.058946 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:24.058804 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:19:24.059497 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:24.059478 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8grmf" Jul 17 21:19:25.220891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:25.220853 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-72zq9" Jul 17 21:19:25.221267 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:25.220923 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-86qlf" Jul 17 21:19:26.100878 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.100847 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv"] Jul 17 21:19:26.105165 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.105142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.108040 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.108013 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 21:19:26.108153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.108045 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 21:19:26.108153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.108058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\"" Jul 17 21:19:26.109086 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.109069 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 21:19:26.113265 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.113243 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv"] Jul 17 21:19:26.233703 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.234095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.234095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9hz\" (UniqueName: \"kubernetes.io/projected/e532825c-cf85-4dc0-804e-bdf30c63ce8e-kube-api-access-hj9hz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.234095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.234095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.234095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.233971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335135 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9hz\" (UniqueName: \"kubernetes.io/projected/e532825c-cf85-4dc0-804e-bdf30c63ce8e-kube-api-access-hj9hz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335592 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335670 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.335897 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.335872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.337613 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.337596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e532825c-cf85-4dc0-804e-bdf30c63ce8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.337891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.337875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e532825c-cf85-4dc0-804e-bdf30c63ce8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.343471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.343448 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9hz\" (UniqueName: \"kubernetes.io/projected/e532825c-cf85-4dc0-804e-bdf30c63ce8e-kube-api-access-hj9hz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv\" (UID: \"e532825c-cf85-4dc0-804e-bdf30c63ce8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.416314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.416101 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:26.549968 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.549945 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv"] Jul 17 21:19:26.551589 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:26.551563 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode532825c_cf85_4dc0_804e_bdf30c63ce8e.slice/crio-18175c94ddd2ae721f7b563bd47605338c7a5557b72fac72f36ba15cdc2dbe9f WatchSource:0}: Error finding container 18175c94ddd2ae721f7b563bd47605338c7a5557b72fac72f36ba15cdc2dbe9f: Status 404 returned error can't find the container with id 18175c94ddd2ae721f7b563bd47605338c7a5557b72fac72f36ba15cdc2dbe9f Jul 17 21:19:26.552971 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:26.552953 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:19:27.291901 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:27.291869 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" event={"ID":"e532825c-cf85-4dc0-804e-bdf30c63ce8e","Type":"ContainerStarted","Data":"18175c94ddd2ae721f7b563bd47605338c7a5557b72fac72f36ba15cdc2dbe9f"} Jul 17 21:19:28.298382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:28.298350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" event={"ID":"e532825c-cf85-4dc0-804e-bdf30c63ce8e","Type":"ContainerStarted","Data":"d2282e3860689430e66535bddd32d6ab2f627d106987b84e06a921d9db90143f"} Jul 17 21:19:28.316219 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:28.316177 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" podStartSLOduration=1.167383627 podStartE2EDuration="2.31616192s" totalImagesPullingTime="1.148778293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:28.315528075 +0000 UTC m=+641.620527009" watchObservedRunningTime="2026-07-17 21:19:28.31616192 +0000 UTC m=+641.621160890" Jul 17 21:19:36.416345 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:36.416311 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:36.416741 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:36.416388 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:36.429370 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:36.429347 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:37.354459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:37.354433 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv" Jul 17 21:19:42.504286 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.504256 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7"] Jul 17 21:19:42.506821 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.506804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.509727 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.509700 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 21:19:42.519016 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.518995 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7"] Jul 17 21:19:42.590671 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.590872 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pf2c\" (UniqueName: \"kubernetes.io/projected/d9067147-c980-4af9-ad04-f4f543d26dd6-kube-api-access-2pf2c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.590872 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.591020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.591020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.591020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.590984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9067147-c980-4af9-ad04-f4f543d26dd6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.691615 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.691615 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9067147-c980-4af9-ad04-f4f543d26dd6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.691891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.691891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pf2c\" (UniqueName: \"kubernetes.io/projected/d9067147-c980-4af9-ad04-f4f543d26dd6-kube-api-access-2pf2c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.691891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.692060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.692060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.692060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.691989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.692226 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.692133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.694116 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.694094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9067147-c980-4af9-ad04-f4f543d26dd6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.694292 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.694274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9067147-c980-4af9-ad04-f4f543d26dd6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.700166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.700146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pf2c\" (UniqueName: \"kubernetes.io/projected/d9067147-c980-4af9-ad04-f4f543d26dd6-kube-api-access-2pf2c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5rwn7\" (UID: \"d9067147-c980-4af9-ad04-f4f543d26dd6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.817389 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.817299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:42.945523 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:42.945499 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7"] Jul 17 21:19:42.947294 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:42.947271 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9067147_c980_4af9_ad04_f4f543d26dd6.slice/crio-41ce2d7aca202d0c511cbd9310d75f28f6abf64ff7e79b4bbbf07ca30e1326cb WatchSource:0}: Error finding container 41ce2d7aca202d0c511cbd9310d75f28f6abf64ff7e79b4bbbf07ca30e1326cb: Status 404 returned error can't find the container with id 41ce2d7aca202d0c511cbd9310d75f28f6abf64ff7e79b4bbbf07ca30e1326cb Jul 17 21:19:43.373804 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:43.373690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" event={"ID":"d9067147-c980-4af9-ad04-f4f543d26dd6","Type":"ContainerStarted","Data":"9ce1f522a9f0a442e12441a93a4256b25dcb6991839e3fbf71de8fa62f57647a"} Jul 17 21:19:43.373804 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:43.373760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" event={"ID":"d9067147-c980-4af9-ad04-f4f543d26dd6","Type":"ContainerStarted","Data":"41ce2d7aca202d0c511cbd9310d75f28f6abf64ff7e79b4bbbf07ca30e1326cb"} Jul 17 21:19:43.394286 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:43.394233 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" podStartSLOduration=1.226850398 podStartE2EDuration="1.394217163s" totalImagesPullingTime="167.366765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:43.392257522 +0000 UTC m=+656.697256453" watchObservedRunningTime="2026-07-17 21:19:43.394217163 +0000 UTC m=+656.699216095" Jul 17 21:19:47.231225 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.231190 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg"] Jul 17 21:19:47.236841 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.236816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.239726 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.239705 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 17 21:19:47.245253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.245104 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg"] Jul 17 21:19:47.334494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.334494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d099393b-490d-4d98-ac6d-56197b0fc90b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.334692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.334807 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.334871 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqpl\" (UniqueName: \"kubernetes.io/projected/d099393b-490d-4d98-ac6d-56197b0fc90b-kube-api-access-vwqpl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.334921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.334906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.435921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.435883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.435990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqpl\" (UniqueName: \"kubernetes.io/projected/d099393b-490d-4d98-ac6d-56197b0fc90b-kube-api-access-vwqpl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436274 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436274 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d099393b-490d-4d98-ac6d-56197b0fc90b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.436633 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.436612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.439039 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.438984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d099393b-490d-4d98-ac6d-56197b0fc90b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.439277 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.439261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d099393b-490d-4d98-ac6d-56197b0fc90b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.451960 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.451939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqpl\" (UniqueName: \"kubernetes.io/projected/d099393b-490d-4d98-ac6d-56197b0fc90b-kube-api-access-vwqpl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg\" (UID: \"d099393b-490d-4d98-ac6d-56197b0fc90b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.549442 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.549411 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:47.689527 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:47.689478 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg"] Jul 17 21:19:47.691495 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:19:47.691467 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd099393b_490d_4d98_ac6d_56197b0fc90b.slice/crio-7340b308b555e30e09ba141d6c6928b8ab250801af7fd23723e79c390c8d1e01 WatchSource:0}: Error finding container 7340b308b555e30e09ba141d6c6928b8ab250801af7fd23723e79c390c8d1e01: Status 404 returned error can't find the container with id 7340b308b555e30e09ba141d6c6928b8ab250801af7fd23723e79c390c8d1e01 Jul 17 21:19:48.401312 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:48.401279 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" event={"ID":"d099393b-490d-4d98-ac6d-56197b0fc90b","Type":"ContainerStarted","Data":"009327de60938ad9eb5dc6e8e42a72b6d8d594f9c0fba11f4b6d57c69f3852c6"} Jul 17 21:19:48.401682 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:48.401330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" event={"ID":"d099393b-490d-4d98-ac6d-56197b0fc90b","Type":"ContainerStarted","Data":"7340b308b555e30e09ba141d6c6928b8ab250801af7fd23723e79c390c8d1e01"} Jul 17 21:19:48.419572 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:48.419525 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" podStartSLOduration=1.251017838 podStartE2EDuration="1.419508872s" totalImagesPullingTime="168.491034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:48.41873004 +0000 UTC m=+661.723728971" watchObservedRunningTime="2026-07-17 21:19:48.419508872 +0000 UTC m=+661.724507802" Jul 17 21:19:52.817519 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:52.817479 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:52.817948 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:52.817549 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:52.830181 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:52.830159 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:53.435319 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:53.435286 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5rwn7" Jul 17 21:19:57.549942 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:57.549910 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:57.550433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:57.549997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:57.562465 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:57.562445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:19:58.461060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:19:58.461031 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg" Jul 17 21:22:17.559235 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.559131 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:22:17.559703 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.559421 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-557f54c68b-p2rvk" podUID="b01b9964-da50-4279-be4f-e4c9a696cbb8" containerName="manager" containerID="cri-o://0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38" gracePeriod=30 Jul 17 21:22:17.810140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.810082 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:22:17.873399 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.873367 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m72lc\" (UniqueName: \"kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc\") pod \"b01b9964-da50-4279-be4f-e4c9a696cbb8\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " Jul 17 21:22:17.873582 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.873494 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert\") pod \"b01b9964-da50-4279-be4f-e4c9a696cbb8\" (UID: \"b01b9964-da50-4279-be4f-e4c9a696cbb8\") " Jul 17 21:22:17.875536 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.875501 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b01b9964-da50-4279-be4f-e4c9a696cbb8" (UID: "b01b9964-da50-4279-be4f-e4c9a696cbb8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:22:17.875654 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.875606 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc" (OuterVolumeSpecName: "kube-api-access-m72lc") pod "b01b9964-da50-4279-be4f-e4c9a696cbb8" (UID: "b01b9964-da50-4279-be4f-e4c9a696cbb8"). InnerVolumeSpecName "kube-api-access-m72lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:22:17.975182 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.975146 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01b9964-da50-4279-be4f-e4c9a696cbb8-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:22:17.975182 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:17.975175 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72lc\" (UniqueName: \"kubernetes.io/projected/b01b9964-da50-4279-be4f-e4c9a696cbb8-kube-api-access-m72lc\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:22:18.146022 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.145930 2573 generic.go:356] "Generic (PLEG): container finished" podID="b01b9964-da50-4279-be4f-e4c9a696cbb8" containerID="0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38" exitCode=0 Jul 17 21:22:18.146022 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.145972 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-p2rvk" Jul 17 21:22:18.146244 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.146020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-p2rvk" event={"ID":"b01b9964-da50-4279-be4f-e4c9a696cbb8","Type":"ContainerDied","Data":"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38"} Jul 17 21:22:18.146244 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.146068 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-p2rvk" event={"ID":"b01b9964-da50-4279-be4f-e4c9a696cbb8","Type":"ContainerDied","Data":"587e3ceee21b1679ef2f39b31cbf287961cf1bf5fcd2c6f22d06ed617b972d0a"} Jul 17 21:22:18.146244 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.146090 2573 scope.go:117] "RemoveContainer" containerID="0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38" Jul 17 21:22:18.155278 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.155260 2573 scope.go:117] "RemoveContainer" containerID="0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38" Jul 17 21:22:18.155546 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:22:18.155525 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38\": container with ID starting with 0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38 not found: ID does not exist" containerID="0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38" Jul 17 21:22:18.155631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.155557 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38"} err="failed to get container status \"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38\": rpc error: code = NotFound desc = could not find container \"0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38\": container with ID starting with 0ece4ddb60797d1220053a095e26575ec7b4ae23629ea54e511321f71a0c6d38 not found: ID does not exist" Jul 17 21:22:18.167024 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.166997 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:22:18.173019 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:18.172999 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-557f54c68b-p2rvk"] Jul 17 21:22:19.238787 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:22:19.238739 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01b9964-da50-4279-be4f-e4c9a696cbb8" path="/var/lib/kubelet/pods/b01b9964-da50-4279-be4f-e4c9a696cbb8/volumes" Jul 17 21:23:18.956414 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.956379 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:18.956973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.956827 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01b9964-da50-4279-be4f-e4c9a696cbb8" containerName="manager" Jul 17 21:23:18.956973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.956839 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01b9964-da50-4279-be4f-e4c9a696cbb8" containerName="manager" Jul 17 21:23:18.956973 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.956904 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01b9964-da50-4279-be4f-e4c9a696cbb8" containerName="manager" Jul 17 21:23:18.959177 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.959158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:18.962129 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.962110 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:23:18.963327 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.963309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:23:18.963393 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.963350 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ntt\"" Jul 17 21:23:18.972194 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:18.972162 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:19.055674 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.055624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.055890 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.055748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fplq\" (UniqueName: \"kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.157149 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.157116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.157322 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.157189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fplq\" (UniqueName: \"kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.159663 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.159630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.165579 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.165556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fplq\" (UniqueName: \"kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq\") pod \"maas-controller-557f54c68b-c2mbj\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.270828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.270800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:19.404433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.404401 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:19.406204 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:23:19.406175 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82532b88_0c42_4c6b_b5c3_c4c0ae156da2.slice/crio-b4c880e5a049561e00be9e9615d007d5bee20bfb614b45864f821d62c15eea7e WatchSource:0}: Error finding container b4c880e5a049561e00be9e9615d007d5bee20bfb614b45864f821d62c15eea7e: Status 404 returned error can't find the container with id b4c880e5a049561e00be9e9615d007d5bee20bfb614b45864f821d62c15eea7e Jul 17 21:23:19.452389 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:19.452356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-c2mbj" event={"ID":"82532b88-0c42-4c6b-b5c3-c4c0ae156da2","Type":"ContainerStarted","Data":"b4c880e5a049561e00be9e9615d007d5bee20bfb614b45864f821d62c15eea7e"} Jul 17 21:23:20.459360 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:20.459326 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-c2mbj" event={"ID":"82532b88-0c42-4c6b-b5c3-c4c0ae156da2","Type":"ContainerStarted","Data":"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584"} Jul 17 21:23:20.459734 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:20.459447 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:20.491113 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:20.491042 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-557f54c68b-c2mbj" podStartSLOduration=2.088361481 podStartE2EDuration="2.491027361s" totalImagesPullingTime="402.66588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:20.490332482 +0000 UTC m=+873.795331414" watchObservedRunningTime="2026-07-17 21:23:20.491027361 +0000 UTC m=+873.796026292" Jul 17 21:23:31.466681 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:31.466650 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:36.120834 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.120800 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:36.121290 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.121016 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-557f54c68b-c2mbj" podUID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" containerName="manager" containerID="cri-o://537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584" gracePeriod=30 Jul 17 21:23:36.368258 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.368227 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:36.424974 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.424880 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fplq\" (UniqueName: \"kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq\") pod \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " Jul 17 21:23:36.424974 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.424967 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert\") pod \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\" (UID: \"82532b88-0c42-4c6b-b5c3-c4c0ae156da2\") " Jul 17 21:23:36.427573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.427543 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "82532b88-0c42-4c6b-b5c3-c4c0ae156da2" (UID: "82532b88-0c42-4c6b-b5c3-c4c0ae156da2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:23:36.427573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.427556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq" (OuterVolumeSpecName: "kube-api-access-2fplq") pod "82532b88-0c42-4c6b-b5c3-c4c0ae156da2" (UID: "82532b88-0c42-4c6b-b5c3-c4c0ae156da2"). InnerVolumeSpecName "kube-api-access-2fplq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:23:36.526679 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.526649 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fplq\" (UniqueName: \"kubernetes.io/projected/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-kube-api-access-2fplq\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:23:36.526679 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.526675 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82532b88-0c42-4c6b-b5c3-c4c0ae156da2-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:23:36.540082 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.540054 2573 generic.go:356] "Generic (PLEG): container finished" podID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" containerID="537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584" exitCode=0 Jul 17 21:23:36.540202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.540113 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-c2mbj" Jul 17 21:23:36.540202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.540139 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-c2mbj" event={"ID":"82532b88-0c42-4c6b-b5c3-c4c0ae156da2","Type":"ContainerDied","Data":"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584"} Jul 17 21:23:36.540202 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.540195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-c2mbj" event={"ID":"82532b88-0c42-4c6b-b5c3-c4c0ae156da2","Type":"ContainerDied","Data":"b4c880e5a049561e00be9e9615d007d5bee20bfb614b45864f821d62c15eea7e"} Jul 17 21:23:36.540366 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.540219 2573 scope.go:117] "RemoveContainer" containerID="537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584" Jul 17 21:23:36.549573 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.549553 2573 scope.go:117] "RemoveContainer" containerID="537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584" Jul 17 21:23:36.549820 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:23:36.549803 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584\": container with ID starting with 537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584 not found: ID does not exist" containerID="537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584" Jul 17 21:23:36.549881 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.549827 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584"} err="failed to get container status \"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584\": rpc error: code = NotFound desc = could not find container \"537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584\": container with ID starting with 537a7cbdd541c83031ed5d07b05f41d2ffd6c94181a7c13c842e6cfcdaf26584 not found: ID does not exist" Jul 17 21:23:36.568414 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.568374 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:36.570301 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:36.570273 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-557f54c68b-c2mbj"] Jul 17 21:23:37.238023 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.237992 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" path="/var/lib/kubelet/pods/82532b88-0c42-4c6b-b5c3-c4c0ae156da2/volumes" Jul 17 21:23:37.434065 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.434036 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-557f54c68b-8gszz"] Jul 17 21:23:37.434456 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.434443 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" containerName="manager" Jul 17 21:23:37.434510 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.434457 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" containerName="manager" Jul 17 21:23:37.434546 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.434524 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="82532b88-0c42-4c6b-b5c3-c4c0ae156da2" containerName="manager" Jul 17 21:23:37.437388 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.437372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.443110 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.443085 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ntt\"" Jul 17 21:23:37.443267 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.443184 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:23:37.443404 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.443383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:23:37.466385 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.466361 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-8gszz"] Jul 17 21:23:37.535750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.535722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c4644f6-c066-4767-9727-07223eb066f5-webhook-cert\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.535926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.535871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnwf\" (UniqueName: \"kubernetes.io/projected/4c4644f6-c066-4767-9727-07223eb066f5-kube-api-access-2fnwf\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.636381 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.636349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnwf\" (UniqueName: \"kubernetes.io/projected/4c4644f6-c066-4767-9727-07223eb066f5-kube-api-access-2fnwf\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.636381 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.636390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c4644f6-c066-4767-9727-07223eb066f5-webhook-cert\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.639005 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.638984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c4644f6-c066-4767-9727-07223eb066f5-webhook-cert\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.647617 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.647592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnwf\" (UniqueName: \"kubernetes.io/projected/4c4644f6-c066-4767-9727-07223eb066f5-kube-api-access-2fnwf\") pod \"maas-controller-557f54c68b-8gszz\" (UID: \"4c4644f6-c066-4767-9727-07223eb066f5\") " pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.747768 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.747731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:37.882152 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:37.882122 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557f54c68b-8gszz"] Jul 17 21:23:37.883634 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:23:37.883605 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4644f6_c066_4767_9727_07223eb066f5.slice/crio-862141628f27015b50ce575952b2d3bc3ece576153ca89749780937093dbdaf5 WatchSource:0}: Error finding container 862141628f27015b50ce575952b2d3bc3ece576153ca89749780937093dbdaf5: Status 404 returned error can't find the container with id 862141628f27015b50ce575952b2d3bc3ece576153ca89749780937093dbdaf5 Jul 17 21:23:38.559683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:38.559645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-8gszz" event={"ID":"4c4644f6-c066-4767-9727-07223eb066f5","Type":"ContainerStarted","Data":"04de0cae70fa0342ab4b1e683d1cd192295e076437435fb3c22134d70f9a24e0"} Jul 17 21:23:38.560105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:38.559730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557f54c68b-8gszz" event={"ID":"4c4644f6-c066-4767-9727-07223eb066f5","Type":"ContainerStarted","Data":"862141628f27015b50ce575952b2d3bc3ece576153ca89749780937093dbdaf5"} Jul 17 21:23:38.560105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:38.559878 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:23:38.578170 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:38.578130 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-557f54c68b-8gszz" podStartSLOduration=1.131682544 podStartE2EDuration="1.578117599s" totalImagesPullingTime="446.435055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:38.577996962 +0000 UTC m=+891.882995895" watchObservedRunningTime="2026-07-17 21:23:38.578117599 +0000 UTC m=+891.883116530" Jul 17 21:23:47.193095 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:47.193062 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:23:47.193950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:47.193932 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:23:47.227496 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:47.227475 2573 scope.go:117] "RemoveContainer" containerID="2aa3df9f4a60ca030516f4d3a2961bcaf3d76e87d3f231d4136a0ee5ecf102c6" Jul 17 21:23:49.568859 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:23:49.568831 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-557f54c68b-8gszz" Jul 17 21:28:47.229004 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:28:47.228972 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:28:47.230853 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:28:47.230831 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:33:47.263146 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:33:47.263120 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:33:47.266595 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:33:47.266573 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:36:04.815722 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.815678 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:36:04.824282 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.824244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:04.827105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.827082 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-52039111-serving-cert\"" Jul 17 21:36:04.831131 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.831109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:36:04.926457 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.926425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:04.926616 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.926533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:04.926616 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:04.926584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv4h\" (UniqueName: \"kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.027639 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.027607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.027831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.027660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv4h\" (UniqueName: \"kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.027831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.027699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.027913 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:05.027851 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-52039111-serving-cert: secret "maas-api-e2e-ait-52039111-serving-cert" not found Jul 17 21:36:05.027948 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:05.027925 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls podName:1cc4e830-5a77-4a7c-8ab5-49779202cd9a nodeName:}" failed. No retries permitted until 2026-07-17 21:36:05.52790419 +0000 UTC m=+1638.832903099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls") pod "maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" (UID: "1cc4e830-5a77-4a7c-8ab5-49779202cd9a") : secret "maas-api-e2e-ait-52039111-serving-cert" not found Jul 17 21:36:05.028262 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.028244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.039020 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.038999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv4h\" (UniqueName: \"kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.532822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.532745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.535409 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.535382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") pod \"maas-api-e2e-ait-52039111-56dc9b4fbd-75mts\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.736224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.736175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:05.873291 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.873258 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:36:05.874747 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:36:05.874721 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc4e830_5a77_4a7c_8ab5_49779202cd9a.slice/crio-c2fe08a0553ab5a8bd0bc269c33724989c859db5dd884d000df873e9f7e10bad WatchSource:0}: Error finding container c2fe08a0553ab5a8bd0bc269c33724989c859db5dd884d000df873e9f7e10bad: Status 404 returned error can't find the container with id c2fe08a0553ab5a8bd0bc269c33724989c859db5dd884d000df873e9f7e10bad Jul 17 21:36:05.876541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:05.876521 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:36:06.276216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:06.276177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerStarted","Data":"c2fe08a0553ab5a8bd0bc269c33724989c859db5dd884d000df873e9f7e10bad"} Jul 17 21:36:07.282436 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:07.282401 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="033b13f7b0d2e90bf9a1fcda2e7c60ffac0bcf4c35d71f396659e2ce417f8123" exitCode=1 Jul 17 21:36:07.282929 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:07.282444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"033b13f7b0d2e90bf9a1fcda2e7c60ffac0bcf4c35d71f396659e2ce417f8123"} Jul 17 21:36:07.282929 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:07.282763 2573 scope.go:117] "RemoveContainer" containerID="033b13f7b0d2e90bf9a1fcda2e7c60ffac0bcf4c35d71f396659e2ce417f8123" Jul 17 21:36:08.289551 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:08.289514 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" exitCode=1 Jul 17 21:36:08.289950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:08.289598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6"} Jul 17 21:36:08.289950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:08.289653 2573 scope.go:117] "RemoveContainer" containerID="033b13f7b0d2e90bf9a1fcda2e7c60ffac0bcf4c35d71f396659e2ce417f8123" Jul 17 21:36:08.289950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:08.289948 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:08.290209 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:08.290186 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:09.295684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:09.295660 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:09.296083 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:09.295884 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:10.736590 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:10.736550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:10.736996 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:10.736946 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:10.737136 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:10.737115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:15.736716 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:15.736676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:15.737139 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:15.737084 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:15.737275 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:15.737255 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:31.233141 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:31.233110 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:32.415133 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:32.415100 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" exitCode=1 Jul 17 21:36:32.415535 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:32.415179 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614"} Jul 17 21:36:32.415535 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:32.415245 2573 scope.go:117] "RemoveContainer" containerID="8250c6d8cc906a1c9c1f575e970f8c062123b0b4bc5eef873f979bb943790fe6" Jul 17 21:36:32.415642 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:32.415591 2573 scope.go:117] "RemoveContainer" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" Jul 17 21:36:32.415857 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:32.415834 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:35.736705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:35.736661 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:35.737349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:35.736798 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:36:35.737349 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:35.737203 2573 scope.go:117] "RemoveContainer" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" Jul 17 21:36:35.737494 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:35.737466 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:47.236690 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:47.236529 2573 scope.go:117] "RemoveContainer" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" Jul 17 21:36:47.237252 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:36:47.236968 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:36:59.233117 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:36:59.233085 2573 scope.go:117] "RemoveContainer" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" Jul 17 21:37:00.565689 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:00.565657 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" exitCode=1 Jul 17 21:37:00.566162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:00.565730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff"} Jul 17 21:37:00.566162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:00.565818 2573 scope.go:117] "RemoveContainer" containerID="1628f0a302e745e514163c4365915b6bb7a9005f9e2a9a8c4696a888b5181614" Jul 17 21:37:00.566162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:00.566150 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:00.566419 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:00.566395 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:00.736952 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:00.736915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:37:01.576589 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:01.576562 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:01.576985 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:01.576787 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:05.736902 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:05.736864 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:37:05.737304 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:05.737258 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:05.737463 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:05.737442 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:16.232956 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:16.232923 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:16.233413 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:16.233129 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:31.238585 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:31.238558 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:31.238987 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:31.238719 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:46.233134 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:46.233100 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:46.798893 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:46.798863 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerStarted","Data":"ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d"} Jul 17 21:37:46.799105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:46.799084 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:37:46.817387 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:46.817342 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podStartSLOduration=100.492636066 podStartE2EDuration="1m42.817326146s" totalImagesPullingTime="2.32469008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:46.814343802 +0000 UTC m=+1740.119342749" watchObservedRunningTime="2026-07-17 21:37:46.817326146 +0000 UTC m=+1740.122325076" Jul 17 21:37:47.811663 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:47.811625 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" exitCode=1 Jul 17 21:37:47.812162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:47.811762 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d"} Jul 17 21:37:47.812162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:47.811874 2573 scope.go:117] "RemoveContainer" containerID="ba1f53766555a58f61fcfe91578c5691c3956f9091136986721b62c1ad1265ff" Jul 17 21:37:47.812162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:47.812107 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:37:47.812360 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:47.812340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:48.819131 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:48.819107 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:37:48.819519 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:48.819297 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:37:55.737140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:55.737108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:37:55.737608 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:37:55.737505 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:37:55.737700 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:37:55.737678 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:38:06.233197 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:06.233164 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:38:06.233654 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:38:06.233354 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:38:21.233105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:21.233073 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:38:21.233521 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:38:21.233253 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:38:33.233114 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:33.233081 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:38:33.233585 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:38:33.233257 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:38:46.233181 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:46.233145 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:38:46.233611 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:38:46.233327 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:38:47.295473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:47.295443 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:38:47.301957 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:47.301934 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:38:59.232969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:38:59.232932 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:38:59.233433 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:38:59.233272 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:39:11.233191 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:11.233159 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:39:12.246812 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:12.246759 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerID="622209b5ec78d43420acba0eb2e61be12535c789df43ce9c197995418857767f" exitCode=1 Jul 17 21:39:12.247227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:12.246811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"622209b5ec78d43420acba0eb2e61be12535c789df43ce9c197995418857767f"} Jul 17 21:39:12.247227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:12.246872 2573 scope.go:117] "RemoveContainer" containerID="ef392fd76d04249664a4439d844184059b724a69b8aee3a2b24607667d3fe84d" Jul 17 21:39:12.247227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:12.247152 2573 scope.go:117] "RemoveContainer" containerID="622209b5ec78d43420acba0eb2e61be12535c789df43ce9c197995418857767f" Jul 17 21:39:12.247451 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:12.247427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-52039111-56dc9b4fbd-75mts_odh-ai-gateway-infra(1cc4e830-5a77-4a7c-8ab5-49779202cd9a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" Jul 17 21:39:15.092831 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.092756 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:39:15.250261 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.250238 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:39:15.271713 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.271690 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" Jul 17 21:39:15.271939 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.271691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts" event={"ID":"1cc4e830-5a77-4a7c-8ab5-49779202cd9a","Type":"ContainerDied","Data":"c2fe08a0553ab5a8bd0bc269c33724989c859db5dd884d000df873e9f7e10bad"} Jul 17 21:39:15.272106 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.272088 2573 scope.go:117] "RemoveContainer" containerID="622209b5ec78d43420acba0eb2e61be12535c789df43ce9c197995418857767f" Jul 17 21:39:15.316745 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.316714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") pod \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " Jul 17 21:39:15.316924 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.316864 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqv4h\" (UniqueName: \"kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h\") pod \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " Jul 17 21:39:15.316924 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.316884 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca\") pod \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\" (UID: \"1cc4e830-5a77-4a7c-8ab5-49779202cd9a\") " Jul 17 21:39:15.317291 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.317265 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cc4e830-5a77-4a7c-8ab5-49779202cd9a" (UID: "1cc4e830-5a77-4a7c-8ab5-49779202cd9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:39:15.319064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.319039 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cc4e830-5a77-4a7c-8ab5-49779202cd9a" (UID: "1cc4e830-5a77-4a7c-8ab5-49779202cd9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:39:15.319064 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.319057 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h" (OuterVolumeSpecName: "kube-api-access-nqv4h") pod "1cc4e830-5a77-4a7c-8ab5-49779202cd9a" (UID: "1cc4e830-5a77-4a7c-8ab5-49779202cd9a"). InnerVolumeSpecName "kube-api-access-nqv4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:39:15.418660 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.418576 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqv4h\" (UniqueName: \"kubernetes.io/projected/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-kube-api-access-nqv4h\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:39:15.418660 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.418608 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:39:15.418660 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.418619 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc4e830-5a77-4a7c-8ab5-49779202cd9a-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:39:15.608496 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.608457 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:39:15.628945 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:15.628909 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52039111-56dc9b4fbd-75mts"] Jul 17 21:39:17.239075 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:17.239040 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" path="/var/lib/kubelet/pods/1cc4e830-5a77-4a7c-8ab5-49779202cd9a/volumes" Jul 17 21:39:24.416964 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.416924 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:39:24.417618 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417621 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417642 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417650 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417664 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417672 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417681 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.417728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417689 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.418003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417829 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.418003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417845 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.418003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417855 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.418003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417864 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.418003 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.417873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:39:24.421664 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.421642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.424702 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.424681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cc1422e3-serving-cert\"" Jul 17 21:39:24.430199 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.430175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:39:24.509624 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.509591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.509844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.509681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nnj2\" (UniqueName: \"kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.509844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.509730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.611250 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.611220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.611440 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.611277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nnj2\" (UniqueName: \"kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.611495 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.611450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.612044 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.612026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.613904 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.613886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.622580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.622557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nnj2\" (UniqueName: \"kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2\") pod \"maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.734169 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.734071 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:24.878074 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:24.878046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:39:24.879854 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:39:24.879825 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c0a40c_7ac0_492f_8f5c_409e35191c04.slice/crio-fa24e85280a24ac59026dad1dfc4ac131531cd354aff1e11e32ca1ba72ee9cf9 WatchSource:0}: Error finding container fa24e85280a24ac59026dad1dfc4ac131531cd354aff1e11e32ca1ba72ee9cf9: Status 404 returned error can't find the container with id fa24e85280a24ac59026dad1dfc4ac131531cd354aff1e11e32ca1ba72ee9cf9 Jul 17 21:39:25.322127 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:25.322089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerStarted","Data":"2e6c139a4368042d1d59fdaee212e724f8d443175f11292b659f62891cef623a"} Jul 17 21:39:25.322305 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:25.322155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerStarted","Data":"fa24e85280a24ac59026dad1dfc4ac131531cd354aff1e11e32ca1ba72ee9cf9"} Jul 17 21:39:25.322305 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:25.322199 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:25.363322 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:25.363258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podStartSLOduration=1.015546067 podStartE2EDuration="1.363234787s" totalImagesPullingTime="347.68872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:39:25.362014394 +0000 UTC m=+1838.667013344" watchObservedRunningTime="2026-07-17 21:39:25.363234787 +0000 UTC m=+1838.668233722" Jul 17 21:39:26.328641 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:26.328603 2573 generic.go:356] "Generic (PLEG): container finished" podID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerID="2e6c139a4368042d1d59fdaee212e724f8d443175f11292b659f62891cef623a" exitCode=1 Jul 17 21:39:26.329171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:26.328686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerDied","Data":"2e6c139a4368042d1d59fdaee212e724f8d443175f11292b659f62891cef623a"} Jul 17 21:39:26.329171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:26.328918 2573 scope.go:117] "RemoveContainer" containerID="2e6c139a4368042d1d59fdaee212e724f8d443175f11292b659f62891cef623a" Jul 17 21:39:27.336938 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:27.336901 2573 generic.go:356] "Generic (PLEG): container finished" podID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" exitCode=1 Jul 17 21:39:27.337396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:27.336994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerDied","Data":"1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4"} Jul 17 21:39:27.337396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:27.337078 2573 scope.go:117] "RemoveContainer" containerID="2e6c139a4368042d1d59fdaee212e724f8d443175f11292b659f62891cef623a" Jul 17 21:39:27.337396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:27.337294 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:27.337539 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:27.337518 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:28.343942 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:28.343913 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:28.344325 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:28.344113 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:30.322407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:30.322362 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:30.322846 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:30.322752 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:30.322987 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:30.322967 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:34.734987 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:34.734950 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:34.735407 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:34.735377 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:34.735571 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:34.735550 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:45.233104 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:45.233067 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:46.439929 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:46.439833 2573 generic.go:356] "Generic (PLEG): container finished" podID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" exitCode=1 Jul 17 21:39:46.439929 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:46.439895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerDied","Data":"0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a"} Jul 17 21:39:46.440368 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:46.439955 2573 scope.go:117] "RemoveContainer" containerID="1320e6c0ddc8b30f7d6622ce555fd99d41ac36c9ed7233bcd03c1b759d0d3fa4" Jul 17 21:39:46.440429 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:46.440384 2573 scope.go:117] "RemoveContainer" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" Jul 17 21:39:46.440615 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:46.440594 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:50.322451 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:50.322411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:50.322940 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:50.322814 2573 scope.go:117] "RemoveContainer" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" Jul 17 21:39:50.323012 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:50.322993 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:39:54.734579 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:54.734538 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:39:54.734995 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:39:54.734960 2573 scope.go:117] "RemoveContainer" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" Jul 17 21:39:54.735154 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:39:54.735135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:40:10.232620 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:10.232528 2573 scope.go:117] "RemoveContainer" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" Jul 17 21:40:11.575509 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:11.575470 2573 generic.go:356] "Generic (PLEG): container finished" podID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" exitCode=1 Jul 17 21:40:11.575958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:11.575515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerDied","Data":"f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc"} Jul 17 21:40:11.575958 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:11.575575 2573 scope.go:117] "RemoveContainer" containerID="0ee351c74cf6e169ed60f2fe9d4d123b625d5eecb2a4defec66922d6f0ae887a" Jul 17 21:40:11.576072 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:11.575989 2573 scope.go:117] "RemoveContainer" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" Jul 17 21:40:11.576282 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:11.576252 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:40:14.734555 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:14.734514 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:40:14.734976 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:14.734911 2573 scope.go:117] "RemoveContainer" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" Jul 17 21:40:14.735135 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:14.735115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:40:15.323323 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:15.323286 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:40:15.596858 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:15.596739 2573 scope.go:117] "RemoveContainer" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" Jul 17 21:40:15.597039 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:15.596999 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:40:31.233206 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:31.233165 2573 scope.go:117] "RemoveContainer" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" Jul 17 21:40:31.233836 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:31.233371 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5_odh-ai-gateway-infra(68c0a40c-7ac0-492f-8f5c-409e35191c04)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" Jul 17 21:40:33.945709 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:33.940804 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:40:34.092640 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.092609 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:40:34.206723 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.206627 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls\") pod \"68c0a40c-7ac0-492f-8f5c-409e35191c04\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " Jul 17 21:40:34.206723 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.206678 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nnj2\" (UniqueName: \"kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2\") pod \"68c0a40c-7ac0-492f-8f5c-409e35191c04\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " Jul 17 21:40:34.206950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.206865 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca\") pod \"68c0a40c-7ac0-492f-8f5c-409e35191c04\" (UID: \"68c0a40c-7ac0-492f-8f5c-409e35191c04\") " Jul 17 21:40:34.207236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.207210 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68c0a40c-7ac0-492f-8f5c-409e35191c04" (UID: "68c0a40c-7ac0-492f-8f5c-409e35191c04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:40:34.208968 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.208945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68c0a40c-7ac0-492f-8f5c-409e35191c04" (UID: "68c0a40c-7ac0-492f-8f5c-409e35191c04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:40:34.209050 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.209026 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2" (OuterVolumeSpecName: "kube-api-access-4nnj2") pod "68c0a40c-7ac0-492f-8f5c-409e35191c04" (UID: "68c0a40c-7ac0-492f-8f5c-409e35191c04"). InnerVolumeSpecName "kube-api-access-4nnj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:40:34.308080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.308046 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c0a40c-7ac0-492f-8f5c-409e35191c04-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:40:34.308080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.308076 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c0a40c-7ac0-492f-8f5c-409e35191c04-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:40:34.308080 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.308086 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nnj2\" (UniqueName: \"kubernetes.io/projected/68c0a40c-7ac0-492f-8f5c-409e35191c04-kube-api-access-4nnj2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:40:34.692595 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.692565 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" Jul 17 21:40:34.692824 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.692602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5" event={"ID":"68c0a40c-7ac0-492f-8f5c-409e35191c04","Type":"ContainerDied","Data":"fa24e85280a24ac59026dad1dfc4ac131531cd354aff1e11e32ca1ba72ee9cf9"} Jul 17 21:40:34.692824 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.692666 2573 scope.go:117] "RemoveContainer" containerID="f011da7d274d8022618ba107202b17d5b7ebfd3c168f73a30a4b025f5bab65dc" Jul 17 21:40:34.714975 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.714943 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:40:34.718585 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:34.718560 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc1422e3-5b48cdbd66-cm2j5"] Jul 17 21:40:35.239157 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:35.239124 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" path="/var/lib/kubelet/pods/68c0a40c-7ac0-492f-8f5c-409e35191c04/volumes" Jul 17 21:40:41.964687 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.964640 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965138 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965153 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965165 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965170 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965178 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965184 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965191 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965196 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965202 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965207 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965214 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965211 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965219 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965313 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965325 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.965884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965331 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc4e830-5a77-4a7c-8ab5-49779202cd9a" containerName="maas-api" Jul 17 21:40:41.965884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.965339 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:40:41.970111 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.970091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:41.973069 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.973042 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-db4eb08b-serving-cert\"" Jul 17 21:40:41.981162 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:41.981132 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:40:42.088253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.088216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkxp\" (UniqueName: \"kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.088425 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.088303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.088425 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.088342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.189352 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.189314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkxp\" (UniqueName: \"kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.189534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.189396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.189534 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.189435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.189534 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:42.189532 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-serving-cert: secret "maas-api-e2e-ait-db4eb08b-serving-cert" not found Jul 17 21:40:42.189663 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:42.189585 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls podName:7ab8285d-47d4-4862-b0d6-18e85fea5dc4 nodeName:}" failed. No retries permitted until 2026-07-17 21:40:42.689569384 +0000 UTC m=+1915.994568293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls") pod "maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" (UID: "7ab8285d-47d4-4862-b0d6-18e85fea5dc4") : secret "maas-api-e2e-ait-db4eb08b-serving-cert" not found Jul 17 21:40:42.190062 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.190042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.198359 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.198338 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkxp\" (UniqueName: \"kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.693688 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.693649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.696206 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.696177 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") pod \"maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:42.882258 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:42.882218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:43.230381 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:43.230354 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:40:43.232436 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:40:43.232411 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab8285d_47d4_4862_b0d6_18e85fea5dc4.slice/crio-50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf WatchSource:0}: Error finding container 50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf: Status 404 returned error can't find the container with id 50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf Jul 17 21:40:43.740513 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:43.740479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerStarted","Data":"8776ef79375c783fee56d4de8ebf2ee5026e06468a6059d91b588e169f338ae5"} Jul 17 21:40:43.740697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:43.740530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerStarted","Data":"50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf"} Jul 17 21:40:43.740697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:43.740574 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:43.762373 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:43.762309 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podStartSLOduration=2.374169831 podStartE2EDuration="2.762287856s" totalImagesPullingTime="388.118025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:40:43.758138587 +0000 UTC m=+1917.063137519" watchObservedRunningTime="2026-07-17 21:40:43.762287856 +0000 UTC m=+1917.067286788" Jul 17 21:40:44.746873 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:44.746832 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="8776ef79375c783fee56d4de8ebf2ee5026e06468a6059d91b588e169f338ae5" exitCode=1 Jul 17 21:40:44.747299 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:44.746919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"8776ef79375c783fee56d4de8ebf2ee5026e06468a6059d91b588e169f338ae5"} Jul 17 21:40:44.747299 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:44.747228 2573 scope.go:117] "RemoveContainer" containerID="8776ef79375c783fee56d4de8ebf2ee5026e06468a6059d91b588e169f338ae5" Jul 17 21:40:45.753520 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:45.753486 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" exitCode=1 Jul 17 21:40:45.754002 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:45.753580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69"} Jul 17 21:40:45.754002 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:45.753650 2573 scope.go:117] "RemoveContainer" containerID="8776ef79375c783fee56d4de8ebf2ee5026e06468a6059d91b588e169f338ae5" Jul 17 21:40:45.754002 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:45.753899 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:40:45.754170 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:45.754124 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:40:46.760455 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:46.760422 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:40:46.760863 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:46.760625 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:40:48.740938 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:48.740884 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:48.741381 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:48.741366 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:40:48.741601 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:48.741582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:40:52.883008 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:52.882966 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:40:52.883513 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:40:52.883496 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:40:52.883755 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:40:52.883731 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:05.232908 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:05.232876 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:41:05.857520 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:05.857479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerStarted","Data":"0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9"} Jul 17 21:41:05.857728 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:05.857685 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:41:06.869597 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:06.869562 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" exitCode=1 Jul 17 21:41:06.870038 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:06.869651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9"} Jul 17 21:41:06.870038 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:06.869716 2573 scope.go:117] "RemoveContainer" containerID="fd796bb6cc7fcad41c73b8e2c86b3ec3edb74854dffb9b8ca8e458d27b8bbd69" Jul 17 21:41:06.870038 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:06.870034 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:06.870311 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:06.870287 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:07.876731 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:07.876692 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:07.877172 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:07.876910 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:12.883088 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:12.883046 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:41:12.883506 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:12.883491 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:12.883711 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:12.883689 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:25.232752 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:25.232722 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:25.233156 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:25.232977 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:39.233681 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:39.233604 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:39.234180 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:39.233938 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:41:40.035828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:40.035789 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" exitCode=1 Jul 17 21:41:40.035828 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:40.035815 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0"} Jul 17 21:41:40.036043 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:40.035878 2573 scope.go:117] "RemoveContainer" containerID="0e9c4b9a7107769915c6c687726ee0cf2ea203fe41a3ba2a488399a96f8c3cb9" Jul 17 21:41:40.036206 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:40.036191 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:41:40.036448 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:40.036414 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:40.858720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:40.858680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:41:41.042916 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:41.042888 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:41:41.043100 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:41.043079 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:42.883391 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:42.883350 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:41:42.883827 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:42.883763 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:41:42.883987 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:42.883968 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:41:55.233042 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:41:55.233014 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:41:55.233415 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:41:55.233190 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:08.232664 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:08.232633 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:42:08.233082 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:08.232869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:21.233345 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:21.233309 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:42:22.259837 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:22.259802 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" exitCode=1 Jul 17 21:42:22.260269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:22.259875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87"} Jul 17 21:42:22.260269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:22.259941 2573 scope.go:117] "RemoveContainer" containerID="aa9f260d807d1e14e73831ffa09f738b52c882e3002819432d3e1cf4046fead0" Jul 17 21:42:22.260404 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:22.260302 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:42:22.260555 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:22.260533 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:22.883376 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:22.883337 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:42:23.266669 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:23.266629 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:42:23.267074 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:23.266843 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:25.857901 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:25.857852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:42:25.858310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:25.858249 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:42:25.858514 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:25.858492 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:41.233171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:41.233128 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:42:41.233660 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:41.233299 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:42:52.232893 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:42:52.232861 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:42:52.233383 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:42:52.233069 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:05.233557 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:05.233523 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:05.233961 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:05.233717 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:17.235109 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:17.235081 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:17.235509 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:17.235301 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:28.233343 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:28.233310 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:28.233883 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:28.233483 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:41.232610 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:41.232575 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:41.233024 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:41.232765 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:47.329668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:47.329641 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:43:47.336465 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:47.336444 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:43:52.232972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:52.232937 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:53.726816 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:53.726758 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" exitCode=1 Jul 17 21:43:53.727348 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:53.726815 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff"} Jul 17 21:43:53.727348 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:53.726887 2573 scope.go:117] "RemoveContainer" containerID="a2a842c58790ba779e86e8623509baa2ba24231e60fa4760856d4d6b1f2b6b87" Jul 17 21:43:53.727348 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:53.727315 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:43:53.727598 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:53.727572 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:43:55.857954 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:55.857913 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:43:55.858369 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:43:55.858272 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:43:55.858521 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:43:55.858501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:44:02.883383 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:44:02.883344 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:44:02.883897 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:44:02.883801 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:44:02.884027 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:44:02.884007 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:44:17.235631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:44:17.235602 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:44:17.236024 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:44:17.235854 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:44:32.233224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:44:32.233185 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:44:32.233704 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:44:32.233451 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:44:46.233541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:44:46.233466 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:44:46.234006 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:44:46.233665 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:45:01.232829 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:45:01.232766 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:45:01.233311 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:45:01.233021 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:45:15.239267 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:45:15.239232 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:45:15.239661 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:45:15.239381 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:45:28.233365 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:45:28.233331 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:45:28.233958 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:45:28.233583 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:45:42.233420 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:45:42.233385 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:45:42.233850 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:45:42.233607 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:45:55.232725 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:45:55.232697 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:45:55.233143 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:45:55.232922 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:06.233027 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:06.232998 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:46:06.235275 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:06.233182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:18.232512 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:18.232481 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:46:18.232987 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:18.232737 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:30.233371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:30.233340 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:46:30.233855 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:30.233617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:41.232895 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:41.232862 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:46:41.233382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:41.233155 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:46:42.570200 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:42.570169 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerID="6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14" exitCode=1 Jul 17 21:46:42.570656 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:42.570242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14"} Jul 17 21:46:42.570656 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:42.570316 2573 scope.go:117] "RemoveContainer" containerID="7db31c1371253937c2e978157f7ea8cd1da97552c3dcd4f96b35e72c622632ff" Jul 17 21:46:42.570656 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:42.570626 2573 scope.go:117] "RemoveContainer" containerID="6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14" Jul 17 21:46:42.570871 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:42.570847 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:42.882880 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:42.882760 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:46:43.576493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:43.576463 2573 scope.go:117] "RemoveContainer" containerID="6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14" Jul 17 21:46:43.576869 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:43.576637 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:45.858229 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:45.858188 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:46:45.858631 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:45.858573 2573 scope.go:117] "RemoveContainer" containerID="6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14" Jul 17 21:46:45.858818 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:46:45.858762 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs_odh-ai-gateway-infra(7ab8285d-47d4-4862-b0d6-18e85fea5dc4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" Jul 17 21:46:52.477925 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.477889 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:46:52.623856 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.623816 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" event={"ID":"7ab8285d-47d4-4862-b0d6-18e85fea5dc4","Type":"ContainerDied","Data":"50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf"} Jul 17 21:46:52.624000 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.623891 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cf85ce7700c48f17fbaa3c90f67b2492598a1479df361bb9b49fef0bae4abf" Jul 17 21:46:52.636689 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.636666 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:46:52.746637 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.746612 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca\") pod \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " Jul 17 21:46:52.746835 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.746649 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") pod \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " Jul 17 21:46:52.746835 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.746747 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqkxp\" (UniqueName: \"kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp\") pod \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\" (UID: \"7ab8285d-47d4-4862-b0d6-18e85fea5dc4\") " Jul 17 21:46:52.747004 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.746981 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ab8285d-47d4-4862-b0d6-18e85fea5dc4" (UID: "7ab8285d-47d4-4862-b0d6-18e85fea5dc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:46:52.747089 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.747066 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:46:52.748921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.748897 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp" (OuterVolumeSpecName: "kube-api-access-tqkxp") pod "7ab8285d-47d4-4862-b0d6-18e85fea5dc4" (UID: "7ab8285d-47d4-4862-b0d6-18e85fea5dc4"). InnerVolumeSpecName "kube-api-access-tqkxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:46:52.749004 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.748950 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ab8285d-47d4-4862-b0d6-18e85fea5dc4" (UID: "7ab8285d-47d4-4862-b0d6-18e85fea5dc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:46:52.848554 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.848522 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:46:52.848554 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:52.848547 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqkxp\" (UniqueName: \"kubernetes.io/projected/7ab8285d-47d4-4862-b0d6-18e85fea5dc4-kube-api-access-tqkxp\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:46:53.628291 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:53.628214 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs" Jul 17 21:46:53.646660 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:53.646633 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:46:53.650434 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:53.650411 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db4eb08b-68dcf4845f-gj5zs"] Jul 17 21:46:55.238830 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:46:55.238790 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" path="/var/lib/kubelet/pods/7ab8285d-47d4-4862-b0d6-18e85fea5dc4/volumes" Jul 17 21:47:00.531743 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.531704 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:47:00.532167 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532152 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532169 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532178 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532183 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532194 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532200 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532208 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532213 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532221 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532226 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532233 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532237 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532319 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532326 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532332 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532338 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532344 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c0a40c-7ac0-492f-8f5c-409e35191c04" containerName="maas-api" Jul 17 21:47:00.532435 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.532349 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:47:00.535710 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.535696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.538729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.538706 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-eecee1fd-serving-cert\"" Jul 17 21:47:00.547254 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.547231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:47:00.624303 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.624271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.624473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.624313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.624473 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.624427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.725854 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.725814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.726027 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.725866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.726027 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.725950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.726125 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:00.726106 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-serving-cert: secret "maas-api-e2e-derive-eecee1fd-serving-cert" not found Jul 17 21:47:00.726177 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:00.726168 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls podName:d22e35aa-fa69-4d4c-97ad-cec2fa032b39 nodeName:}" failed. No retries permitted until 2026-07-17 21:47:01.226151797 +0000 UTC m=+2294.531150706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls") pod "maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" (UID: "d22e35aa-fa69-4d4c-97ad-cec2fa032b39") : secret "maas-api-e2e-derive-eecee1fd-serving-cert" not found Jul 17 21:47:00.726507 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.726488 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:00.737209 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:00.737179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:01.231129 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:01.231098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:01.233695 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:01.233664 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") pod \"maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:01.446492 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:01.446461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:01.585736 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:01.585709 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:47:01.587698 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:47:01.587672 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22e35aa_fa69_4d4c_97ad_cec2fa032b39.slice/crio-d80364558f5537e52bb4a5702ea154dbb591e6f017962695e954c1f7cb949785 WatchSource:0}: Error finding container d80364558f5537e52bb4a5702ea154dbb591e6f017962695e954c1f7cb949785: Status 404 returned error can't find the container with id d80364558f5537e52bb4a5702ea154dbb591e6f017962695e954c1f7cb949785 Jul 17 21:47:01.668622 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:01.668587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerStarted","Data":"d80364558f5537e52bb4a5702ea154dbb591e6f017962695e954c1f7cb949785"} Jul 17 21:47:02.675396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:02.675309 2573 generic.go:356] "Generic (PLEG): container finished" podID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerID="dd5387470d68082f02569173da797a8837f819a6f00c9f36a12c2922cdb2e12f" exitCode=1 Jul 17 21:47:02.675396 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:02.675347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"dd5387470d68082f02569173da797a8837f819a6f00c9f36a12c2922cdb2e12f"} Jul 17 21:47:02.675810 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:02.675617 2573 scope.go:117] "RemoveContainer" containerID="dd5387470d68082f02569173da797a8837f819a6f00c9f36a12c2922cdb2e12f" Jul 17 21:47:03.685088 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:03.685006 2573 generic.go:356] "Generic (PLEG): container finished" podID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" exitCode=1 Jul 17 21:47:03.685471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:03.685092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1"} Jul 17 21:47:03.685471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:03.685176 2573 scope.go:117] "RemoveContainer" containerID="dd5387470d68082f02569173da797a8837f819a6f00c9f36a12c2922cdb2e12f" Jul 17 21:47:03.685471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:03.685441 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:03.685673 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:03.685650 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:04.692549 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:04.692526 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:04.692930 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:04.692705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:06.447184 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:06.447154 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:06.447552 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:06.447520 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:06.447721 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:06.447700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:11.447555 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:11.447524 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:11.447961 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:11.447932 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:11.448134 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:11.448113 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:26.232710 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:26.232679 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:26.800485 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:26.800450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerStarted","Data":"85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8"} Jul 17 21:47:26.800668 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:26.800650 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:26.821555 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:26.821493 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podStartSLOduration=25.73413818 podStartE2EDuration="26.821476576s" totalImagesPullingTime="1.087338396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:26.821129982 +0000 UTC m=+2320.126128914" watchObservedRunningTime="2026-07-17 21:47:26.821476576 +0000 UTC m=+2320.126475507" Jul 17 21:47:27.806875 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:27.806840 2573 generic.go:356] "Generic (PLEG): container finished" podID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" exitCode=1 Jul 17 21:47:27.807297 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:27.806925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8"} Jul 17 21:47:27.807297 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:27.806984 2573 scope.go:117] "RemoveContainer" containerID="e4d88020768962af5ddf18871e12ea8cb67fe71bb493ddf4740e7a4babe97db1" Jul 17 21:47:27.807297 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:27.807229 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:47:27.807449 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:27.807432 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:28.814216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:28.814188 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:47:28.814600 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:28.814388 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:31.447013 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:31.446978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:47:31.447397 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:31.447370 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:47:31.447557 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:31.447538 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:46.233099 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:46.233016 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:47:46.233546 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:47:46.233247 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:47:59.233209 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:59.233174 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:47:59.977856 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:59.977814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerStarted","Data":"a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02"} Jul 17 21:47:59.978066 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:47:59.978049 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:48:00.984227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:00.984192 2573 generic.go:356] "Generic (PLEG): container finished" podID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" exitCode=1 Jul 17 21:48:00.984639 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:00.984280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02"} Jul 17 21:48:00.984639 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:00.984333 2573 scope.go:117] "RemoveContainer" containerID="85111c8328ab5d7102025048f9186735fc91260c7445ca0e3ab4814ff34f2cd8" Jul 17 21:48:00.984639 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:00.984526 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:00.984851 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:00.984802 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:01.447589 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:01.447557 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:48:01.990636 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:01.990610 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:01.991034 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:01.990812 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:02.995593 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:02.995566 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:02.996006 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:02.995748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:18.233112 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:18.233080 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:18.233586 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:18.233275 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:30.233532 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:30.233501 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:30.233962 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:30.233672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:42.232865 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:42.232834 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:43.200684 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:43.200653 2573 generic.go:356] "Generic (PLEG): container finished" podID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" exitCode=1 Jul 17 21:48:43.200870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:43.200729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1"} Jul 17 21:48:43.200870 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:43.200804 2573 scope.go:117] "RemoveContainer" containerID="a086d1277f0a3394abf3ed4ad3823a07fa1c5b97a3466a478e6dfb3eae856d02" Jul 17 21:48:43.201142 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:43.201128 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:48:43.201364 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:43.201344 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:44.979091 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:44.979055 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:48:44.979520 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:44.979412 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:48:44.979616 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:44.979598 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:48:47.370346 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:47.370318 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:48:47.377345 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:47.377286 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:48:51.447626 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:51.447588 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:48:51.448147 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:48:51.447989 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:48:51.448191 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:48:51.448177 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:49:05.233197 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:49:05.233163 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:49:05.233559 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:49:05.233348 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:49:19.234016 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:49:19.233983 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:49:19.234418 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:49:19.234135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:49:34.233277 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:49:34.233242 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:49:34.233756 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:49:34.233528 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:49:49.238972 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:49:49.238946 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:49:49.239351 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:49:49.239127 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:50:00.232913 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:00.232879 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:50:00.233287 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:50:00.233074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph_odh-ai-gateway-infra(d22e35aa-fa69-4d4c-97ad-cec2fa032b39)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" Jul 17 21:50:09.990249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:09.990208 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:50:10.133546 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.133522 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:50:10.257315 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.257285 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca\") pod \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " Jul 17 21:50:10.257498 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.257358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") pod \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " Jul 17 21:50:10.257498 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.257490 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb\") pod \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\" (UID: \"d22e35aa-fa69-4d4c-97ad-cec2fa032b39\") " Jul 17 21:50:10.257712 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.257681 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d22e35aa-fa69-4d4c-97ad-cec2fa032b39" (UID: "d22e35aa-fa69-4d4c-97ad-cec2fa032b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:50:10.257852 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.257803 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:50:10.259591 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.259569 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d22e35aa-fa69-4d4c-97ad-cec2fa032b39" (UID: "d22e35aa-fa69-4d4c-97ad-cec2fa032b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:50:10.259591 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.259571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb" (OuterVolumeSpecName: "kube-api-access-5jmxb") pod "d22e35aa-fa69-4d4c-97ad-cec2fa032b39" (UID: "d22e35aa-fa69-4d4c-97ad-cec2fa032b39"). InnerVolumeSpecName "kube-api-access-5jmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:50:10.359472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.359440 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:50:10.359472 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.359467 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/d22e35aa-fa69-4d4c-97ad-cec2fa032b39-kube-api-access-5jmxb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:50:10.640846 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.640745 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" Jul 17 21:50:10.640846 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.640790 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph" event={"ID":"d22e35aa-fa69-4d4c-97ad-cec2fa032b39","Type":"ContainerDied","Data":"d80364558f5537e52bb4a5702ea154dbb591e6f017962695e954c1f7cb949785"} Jul 17 21:50:10.640846 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.640846 2573 scope.go:117] "RemoveContainer" containerID="ae6ee48acbd3fc5acd94aa20ba3aac330be216bca28c63b7bb4395a0f8e53ea1" Jul 17 21:50:10.662224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.662196 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:50:10.667890 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:10.667868 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eecee1fd-6cc4dd66cc-nnkph"] Jul 17 21:50:11.238513 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:50:11.238477 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" path="/var/lib/kubelet/pods/d22e35aa-fa69-4d4c-97ad-cec2fa032b39/volumes" Jul 17 21:51:55.021297 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021263 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021803 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021830 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021841 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021851 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021878 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021888 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021899 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021905 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021921 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.021929 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.022045 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.022058 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:51:55.022692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.022071 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.022079 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.022692 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.022090 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:51:55.025375 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.025355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.028231 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.028207 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-62f6c7d6-serving-cert\"" Jul 17 21:51:55.037645 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.037624 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:51:55.179098 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.179065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.179098 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.179100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.179311 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.179149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n966p\" (UniqueName: \"kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.280106 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.280027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n966p\" (UniqueName: \"kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.280253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.280133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.280253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.280158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.280819 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.280763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.282671 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.282651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.288189 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.288168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n966p\" (UniqueName: \"kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p\") pod \"maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.336227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.336205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:55.477908 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.477691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:51:55.480281 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:51:55.480252 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf090985b_1d76_4861_92e8_f945ae53ace6.slice/crio-e52679334c26aaedf56c6a1d099196f8dcbabec7186b3b1736aaee407f7e2f90 WatchSource:0}: Error finding container e52679334c26aaedf56c6a1d099196f8dcbabec7186b3b1736aaee407f7e2f90: Status 404 returned error can't find the container with id e52679334c26aaedf56c6a1d099196f8dcbabec7186b3b1736aaee407f7e2f90 Jul 17 21:51:55.481667 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:55.481649 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:51:56.186848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:56.186739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" event={"ID":"f090985b-1d76-4861-92e8-f945ae53ace6","Type":"ContainerStarted","Data":"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258"} Jul 17 21:51:56.186848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:56.186820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" event={"ID":"f090985b-1d76-4861-92e8-f945ae53ace6","Type":"ContainerStarted","Data":"e52679334c26aaedf56c6a1d099196f8dcbabec7186b3b1736aaee407f7e2f90"} Jul 17 21:51:56.186848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:56.186846 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:51:56.208308 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:51:56.208258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" podStartSLOduration=0.782743571 podStartE2EDuration="1.208241594s" totalImagesPullingTime="425.498023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:56.207273618 +0000 UTC m=+2589.512272550" watchObservedRunningTime="2026-07-17 21:51:56.208241594 +0000 UTC m=+2589.513240525" Jul 17 21:52:02.197222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:02.197190 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:52:16.340329 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.340290 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:52:16.340785 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.340563 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" podUID="f090985b-1d76-4861-92e8-f945ae53ace6" containerName="maas-api" containerID="cri-o://cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258" gracePeriod=30 Jul 17 21:52:16.593191 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.593123 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:52:16.690166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.690134 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls\") pod \"f090985b-1d76-4861-92e8-f945ae53ace6\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " Jul 17 21:52:16.690362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.690207 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n966p\" (UniqueName: \"kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p\") pod \"f090985b-1d76-4861-92e8-f945ae53ace6\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " Jul 17 21:52:16.690362 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.690235 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca\") pod \"f090985b-1d76-4861-92e8-f945ae53ace6\" (UID: \"f090985b-1d76-4861-92e8-f945ae53ace6\") " Jul 17 21:52:16.690629 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.690604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f090985b-1d76-4861-92e8-f945ae53ace6" (UID: "f090985b-1d76-4861-92e8-f945ae53ace6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:52:16.692385 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.692363 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f090985b-1d76-4861-92e8-f945ae53ace6" (UID: "f090985b-1d76-4861-92e8-f945ae53ace6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:52:16.692471 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.692446 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p" (OuterVolumeSpecName: "kube-api-access-n966p") pod "f090985b-1d76-4861-92e8-f945ae53ace6" (UID: "f090985b-1d76-4861-92e8-f945ae53ace6"). InnerVolumeSpecName "kube-api-access-n966p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:52:16.791489 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.791450 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n966p\" (UniqueName: \"kubernetes.io/projected/f090985b-1d76-4861-92e8-f945ae53ace6-kube-api-access-n966p\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:52:16.791489 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.791478 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f090985b-1d76-4861-92e8-f945ae53ace6-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:52:16.791489 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:16.791489 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f090985b-1d76-4861-92e8-f945ae53ace6-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:52:17.294940 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.294891 2573 generic.go:356] "Generic (PLEG): container finished" podID="f090985b-1d76-4861-92e8-f945ae53ace6" containerID="cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258" exitCode=0 Jul 17 21:52:17.294940 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.294933 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" Jul 17 21:52:17.295195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.294952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" event={"ID":"f090985b-1d76-4861-92e8-f945ae53ace6","Type":"ContainerDied","Data":"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258"} Jul 17 21:52:17.295195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.295004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf" event={"ID":"f090985b-1d76-4861-92e8-f945ae53ace6","Type":"ContainerDied","Data":"e52679334c26aaedf56c6a1d099196f8dcbabec7186b3b1736aaee407f7e2f90"} Jul 17 21:52:17.295195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.295033 2573 scope.go:117] "RemoveContainer" containerID="cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258" Jul 17 21:52:17.304153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.304134 2573 scope.go:117] "RemoveContainer" containerID="cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258" Jul 17 21:52:17.304395 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:52:17.304377 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258\": container with ID starting with cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258 not found: ID does not exist" containerID="cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258" Jul 17 21:52:17.304444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.304402 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258"} err="failed to get container status \"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258\": rpc error: code = NotFound desc = could not find container \"cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258\": container with ID starting with cba8abe9769b8785bea170cac32a705761cf12c2aaf1137c200244e438f07258 not found: ID does not exist" Jul 17 21:52:17.311884 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.311856 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:52:17.315523 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:17.315504 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-62f6c7d6-6c796d7ddf-rcsqf"] Jul 17 21:52:19.238697 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:19.238663 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f090985b-1d76-4861-92e8-f945ae53ace6" path="/var/lib/kubelet/pods/f090985b-1d76-4861-92e8-f945ae53ace6/volumes" Jul 17 21:52:47.499669 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:47.499636 2573 scope.go:117] "RemoveContainer" containerID="6a00f60b31ffbbd20a0d8e2d6a6e4ccdd525a9e8f826e448ee1bc87851afda14" Jul 17 21:52:54.245491 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.245439 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:52:54.246120 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246083 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f090985b-1d76-4861-92e8-f945ae53ace6" containerName="maas-api" Jul 17 21:52:54.246120 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246103 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f090985b-1d76-4861-92e8-f945ae53ace6" containerName="maas-api" Jul 17 21:52:54.246280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246134 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:52:54.246280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246142 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:52:54.246280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246248 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22e35aa-fa69-4d4c-97ad-cec2fa032b39" containerName="maas-api" Jul 17 21:52:54.246280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246269 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f090985b-1d76-4861-92e8-f945ae53ace6" containerName="maas-api" Jul 17 21:52:54.246280 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.246280 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8285d-47d4-4862-b0d6-18e85fea5dc4" containerName="maas-api" Jul 17 21:52:54.249547 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.249527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.254224 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.254201 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c39f8651-serving-cert\"" Jul 17 21:52:54.259253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.259228 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:52:54.344350 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.344317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.344527 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.344357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxzw\" (UniqueName: \"kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.344527 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.344406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.444992 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.444963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.445179 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.445001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxzw\" (UniqueName: \"kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.445179 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.445137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.445630 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.445598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.447640 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.447619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.457703 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.457682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxzw\" (UniqueName: \"kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw\") pod \"maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.564183 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.564100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:54.699851 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:54.699825 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:52:54.701977 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:52:54.701950 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f61ee06_21cc_4be5_b4d5_6c0ffb3e67b0.slice/crio-2a4099e6322c9578e7f38d2a1ec596589ed445e47acb2f4f9260a1d6c5a53f87 WatchSource:0}: Error finding container 2a4099e6322c9578e7f38d2a1ec596589ed445e47acb2f4f9260a1d6c5a53f87: Status 404 returned error can't find the container with id 2a4099e6322c9578e7f38d2a1ec596589ed445e47acb2f4f9260a1d6c5a53f87 Jul 17 21:52:55.494261 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:55.494215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" event={"ID":"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0","Type":"ContainerStarted","Data":"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9"} Jul 17 21:52:55.494638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:55.494288 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" event={"ID":"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0","Type":"ContainerStarted","Data":"2a4099e6322c9578e7f38d2a1ec596589ed445e47acb2f4f9260a1d6c5a53f87"} Jul 17 21:52:55.494638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:55.494439 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:52:55.511683 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:52:55.511641 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" podStartSLOduration=1.216964717 podStartE2EDuration="1.511628584s" totalImagesPullingTime="294.663867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:55.509732988 +0000 UTC m=+2648.814731949" watchObservedRunningTime="2026-07-17 21:52:55.511628584 +0000 UTC m=+2648.816627514" Jul 17 21:53:01.504876 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:01.504850 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:53:41.963125 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:41.963036 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:53:41.964013 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:41.963949 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" podUID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" containerName="maas-api" containerID="cri-o://a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9" gracePeriod=30 Jul 17 21:53:42.218608 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.218545 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:53:42.311249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.311221 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca\") pod \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " Jul 17 21:53:42.311443 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.311293 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxzw\" (UniqueName: \"kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw\") pod \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " Jul 17 21:53:42.311443 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.311341 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls\") pod \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\" (UID: \"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0\") " Jul 17 21:53:42.311648 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.311589 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" (UID: "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:53:42.313502 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.313470 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" (UID: "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:53:42.313608 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.313516 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw" (OuterVolumeSpecName: "kube-api-access-tjxzw") pod "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" (UID: "3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0"). InnerVolumeSpecName "kube-api-access-tjxzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:53:42.412538 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.412498 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:53:42.412538 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.412534 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:53:42.412538 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.412544 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxzw\" (UniqueName: \"kubernetes.io/projected/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0-kube-api-access-tjxzw\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:53:42.729177 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.729141 2573 generic.go:356] "Generic (PLEG): container finished" podID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" containerID="a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9" exitCode=0 Jul 17 21:53:42.729382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.729181 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" Jul 17 21:53:42.729382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.729226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" event={"ID":"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0","Type":"ContainerDied","Data":"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9"} Jul 17 21:53:42.729382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.729283 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct" event={"ID":"3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0","Type":"ContainerDied","Data":"2a4099e6322c9578e7f38d2a1ec596589ed445e47acb2f4f9260a1d6c5a53f87"} Jul 17 21:53:42.729382 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.729312 2573 scope.go:117] "RemoveContainer" containerID="a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9" Jul 17 21:53:42.739165 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.739148 2573 scope.go:117] "RemoveContainer" containerID="a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9" Jul 17 21:53:42.739418 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:53:42.739401 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9\": container with ID starting with a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9 not found: ID does not exist" containerID="a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9" Jul 17 21:53:42.739463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.739427 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9"} err="failed to get container status \"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9\": rpc error: code = NotFound desc = could not find container \"a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9\": container with ID starting with a7afa37eeeb1f2f8381988630e5310dd09c39a1a210de9f3fe2145170d8342e9 not found: ID does not exist" Jul 17 21:53:42.752070 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.752046 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:53:42.757058 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:42.757038 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c39f8651-664dfb66d4-vxzct"] Jul 17 21:53:43.239032 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:43.239005 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" path="/var/lib/kubelet/pods/3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0/volumes" Jul 17 21:53:47.405056 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:47.405020 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:53:47.414232 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:53:47.414210 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:55:35.946393 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.946353 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:55:35.947088 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.947065 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" containerName="maas-api" Jul 17 21:55:35.947188 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.947101 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" containerName="maas-api" Jul 17 21:55:35.947329 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.947216 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f61ee06-21cc-4be5-b4d5-6c0ffb3e67b0" containerName="maas-api" Jul 17 21:55:35.950423 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.950402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:35.953227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.953206 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ffd161-serving-cert\"" Jul 17 21:55:35.963187 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:35.963163 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:55:36.117292 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.117253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.117474 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.117306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.117474 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.117423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szsqx\" (UniqueName: \"kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.218687 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.218602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsqx\" (UniqueName: \"kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.218687 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.218671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.218921 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.218708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.219444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.219417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.221426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.221404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.233464 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.233439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsqx\" (UniqueName: \"kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx\") pod \"maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.261388 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.261366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:36.605012 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:36.604991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:55:36.607304 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:55:36.607278 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770ad5e1_4449_4ca2_b2b3_c1c270f16e59.slice/crio-788250bd51f23e7961523e31a0c99d937744660bf7bdd936cb270e1ce4c769ab WatchSource:0}: Error finding container 788250bd51f23e7961523e31a0c99d937744660bf7bdd936cb270e1ce4c769ab: Status 404 returned error can't find the container with id 788250bd51f23e7961523e31a0c99d937744660bf7bdd936cb270e1ce4c769ab Jul 17 21:55:37.300154 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:37.300120 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" event={"ID":"770ad5e1-4449-4ca2-b2b3-c1c270f16e59","Type":"ContainerStarted","Data":"788250bd51f23e7961523e31a0c99d937744660bf7bdd936cb270e1ce4c769ab"} Jul 17 21:55:38.306525 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:38.306485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" event={"ID":"770ad5e1-4449-4ca2-b2b3-c1c270f16e59","Type":"ContainerStarted","Data":"b31b85704691317ba9fb5d7a3cc7ce53fd013cef859d5c1d261040aff3e0a878"} Jul 17 21:55:38.306937 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:38.306572 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:38.323190 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:38.323142 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" podStartSLOduration=2.4202213009999998 podStartE2EDuration="3.323127102s" totalImagesPullingTime="902.905801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:38.322309659 +0000 UTC m=+2811.627308592" watchObservedRunningTime="2026-07-17 21:55:38.323127102 +0000 UTC m=+2811.628126033" Jul 17 21:55:44.088216 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.088183 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:55:44.091914 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.091891 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.096045 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.096024 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-786043-serving-cert\"" Jul 17 21:55:44.096555 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.096526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvm7\" (UniqueName: \"kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.096638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.096575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.096638 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.096613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.120823 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.120799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:55:44.197221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.197186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvm7\" (UniqueName: \"kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.197221 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.197225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.197433 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.197271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.198005 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.197987 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.199866 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.199847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.235832 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.235807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvm7\" (UniqueName: \"kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7\") pod \"maas-api-e2e-inf-b-786043-687df99cd4-6qm6w\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.316399 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.316372 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:55:44.402625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.402548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:44.584235 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:44.584209 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:55:44.585588 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:55:44.585558 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31250d65_693c_4832_b5d8_b05e026447eb.slice/crio-10814e053ee307fc6ee4bca6a31fbda6a58bc1b9d0f3544ae6fe4ca895fa1e55 WatchSource:0}: Error finding container 10814e053ee307fc6ee4bca6a31fbda6a58bc1b9d0f3544ae6fe4ca895fa1e55: Status 404 returned error can't find the container with id 10814e053ee307fc6ee4bca6a31fbda6a58bc1b9d0f3544ae6fe4ca895fa1e55 Jul 17 21:55:45.343877 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:45.343837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" event={"ID":"31250d65-693c-4832-b5d8-b05e026447eb","Type":"ContainerStarted","Data":"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72"} Jul 17 21:55:45.344307 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:45.343900 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:55:45.344307 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:45.343916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" event={"ID":"31250d65-693c-4832-b5d8-b05e026447eb","Type":"ContainerStarted","Data":"10814e053ee307fc6ee4bca6a31fbda6a58bc1b9d0f3544ae6fe4ca895fa1e55"} Jul 17 21:55:45.381085 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:45.380992 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" podStartSLOduration=0.836580513 podStartE2EDuration="1.380976291s" totalImagesPullingTime="544.395778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:45.380156274 +0000 UTC m=+2818.685155230" watchObservedRunningTime="2026-07-17 21:55:45.380976291 +0000 UTC m=+2818.685975221" Jul 17 21:55:51.355606 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:55:51.355573 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:57:18.639111 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.639074 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:57:18.639630 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.639333 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" podUID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" containerName="maas-api" containerID="cri-o://b31b85704691317ba9fb5d7a3cc7ce53fd013cef859d5c1d261040aff3e0a878" gracePeriod=30 Jul 17 21:57:18.821388 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.821355 2573 generic.go:356] "Generic (PLEG): container finished" podID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" containerID="b31b85704691317ba9fb5d7a3cc7ce53fd013cef859d5c1d261040aff3e0a878" exitCode=0 Jul 17 21:57:18.821535 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.821427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" event={"ID":"770ad5e1-4449-4ca2-b2b3-c1c270f16e59","Type":"ContainerDied","Data":"b31b85704691317ba9fb5d7a3cc7ce53fd013cef859d5c1d261040aff3e0a878"} Jul 17 21:57:18.890236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.890182 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:57:18.996725 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.996692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca\") pod \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " Jul 17 21:57:18.996926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.996753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls\") pod \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " Jul 17 21:57:18.996926 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.996835 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szsqx\" (UniqueName: \"kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx\") pod \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\" (UID: \"770ad5e1-4449-4ca2-b2b3-c1c270f16e59\") " Jul 17 21:57:18.997164 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.997135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "770ad5e1-4449-4ca2-b2b3-c1c270f16e59" (UID: "770ad5e1-4449-4ca2-b2b3-c1c270f16e59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:57:18.997284 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.997212 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:18.998950 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.998930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "770ad5e1-4449-4ca2-b2b3-c1c270f16e59" (UID: "770ad5e1-4449-4ca2-b2b3-c1c270f16e59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:57:18.999068 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:18.999046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx" (OuterVolumeSpecName: "kube-api-access-szsqx") pod "770ad5e1-4449-4ca2-b2b3-c1c270f16e59" (UID: "770ad5e1-4449-4ca2-b2b3-c1c270f16e59"). InnerVolumeSpecName "kube-api-access-szsqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:57:19.098565 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.098527 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:19.098565 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.098558 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szsqx\" (UniqueName: \"kubernetes.io/projected/770ad5e1-4449-4ca2-b2b3-c1c270f16e59-kube-api-access-szsqx\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:19.827686 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.827655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" event={"ID":"770ad5e1-4449-4ca2-b2b3-c1c270f16e59","Type":"ContainerDied","Data":"788250bd51f23e7961523e31a0c99d937744660bf7bdd936cb270e1ce4c769ab"} Jul 17 21:57:19.828103 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.827702 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf" Jul 17 21:57:19.828103 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.827718 2573 scope.go:117] "RemoveContainer" containerID="b31b85704691317ba9fb5d7a3cc7ce53fd013cef859d5c1d261040aff3e0a878" Jul 17 21:57:19.845453 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.845427 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:57:19.852284 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:19.852261 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffd161-7866b565df-5qgcf"] Jul 17 21:57:21.238422 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:21.238388 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" path="/var/lib/kubelet/pods/770ad5e1-4449-4ca2-b2b3-c1c270f16e59/volumes" Jul 17 21:57:44.789410 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:44.789371 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:57:44.789880 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:44.789642 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" podUID="31250d65-693c-4832-b5d8-b05e026447eb" containerName="maas-api" containerID="cri-o://c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72" gracePeriod=30 Jul 17 21:57:45.242328 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.242292 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:57:45.253170 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.253140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqvm7\" (UniqueName: \"kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7\") pod \"31250d65-693c-4832-b5d8-b05e026447eb\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " Jul 17 21:57:45.253298 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.253187 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca\") pod \"31250d65-693c-4832-b5d8-b05e026447eb\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " Jul 17 21:57:45.253341 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.253325 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls\") pod \"31250d65-693c-4832-b5d8-b05e026447eb\" (UID: \"31250d65-693c-4832-b5d8-b05e026447eb\") " Jul 17 21:57:45.253599 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.253570 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31250d65-693c-4832-b5d8-b05e026447eb" (UID: "31250d65-693c-4832-b5d8-b05e026447eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:57:45.253729 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.253631 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31250d65-693c-4832-b5d8-b05e026447eb-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:45.255342 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.255320 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7" (OuterVolumeSpecName: "kube-api-access-hqvm7") pod "31250d65-693c-4832-b5d8-b05e026447eb" (UID: "31250d65-693c-4832-b5d8-b05e026447eb"). InnerVolumeSpecName "kube-api-access-hqvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:57:45.255512 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.255496 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31250d65-693c-4832-b5d8-b05e026447eb" (UID: "31250d65-693c-4832-b5d8-b05e026447eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:57:45.354275 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.354182 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31250d65-693c-4832-b5d8-b05e026447eb-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:45.354275 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.354212 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvm7\" (UniqueName: \"kubernetes.io/projected/31250d65-693c-4832-b5d8-b05e026447eb-kube-api-access-hqvm7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:57:45.955351 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.955312 2573 generic.go:356] "Generic (PLEG): container finished" podID="31250d65-693c-4832-b5d8-b05e026447eb" containerID="c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72" exitCode=0 Jul 17 21:57:45.955908 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.955358 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" event={"ID":"31250d65-693c-4832-b5d8-b05e026447eb","Type":"ContainerDied","Data":"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72"} Jul 17 21:57:45.956037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.955394 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" Jul 17 21:57:45.956037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.955972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w" event={"ID":"31250d65-693c-4832-b5d8-b05e026447eb","Type":"ContainerDied","Data":"10814e053ee307fc6ee4bca6a31fbda6a58bc1b9d0f3544ae6fe4ca895fa1e55"} Jul 17 21:57:45.956037 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.956031 2573 scope.go:117] "RemoveContainer" containerID="c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72" Jul 17 21:57:45.967418 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.967400 2573 scope.go:117] "RemoveContainer" containerID="c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72" Jul 17 21:57:45.967693 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:57:45.967675 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72\": container with ID starting with c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72 not found: ID does not exist" containerID="c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72" Jul 17 21:57:45.967750 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.967701 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72"} err="failed to get container status \"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72\": rpc error: code = NotFound desc = could not find container \"c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72\": container with ID starting with c2e2115f5b7bc19db196bb97820bac4ed383fa74d990c7cc8c80b91a28c1df72 not found: ID does not exist" Jul 17 21:57:45.981402 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.981372 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:57:45.984621 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:45.984599 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-786043-687df99cd4-6qm6w"] Jul 17 21:57:47.239625 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:57:47.239594 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31250d65-693c-4832-b5d8-b05e026447eb" path="/var/lib/kubelet/pods/31250d65-693c-4832-b5d8-b05e026447eb/volumes" Jul 17 21:58:01.068632 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.068599 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:01.069084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069059 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" containerName="maas-api" Jul 17 21:58:01.069084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069079 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" containerName="maas-api" Jul 17 21:58:01.069160 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069089 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31250d65-693c-4832-b5d8-b05e026447eb" containerName="maas-api" Jul 17 21:58:01.069160 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069094 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31250d65-693c-4832-b5d8-b05e026447eb" containerName="maas-api" Jul 17 21:58:01.069222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069187 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31250d65-693c-4832-b5d8-b05e026447eb" containerName="maas-api" Jul 17 21:58:01.069222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.069202 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="770ad5e1-4449-4ca2-b2b3-c1c270f16e59" containerName="maas-api" Jul 17 21:58:01.072387 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.072369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.075073 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.075054 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-380122-openshift-default-dockercfg-7rztn\"" Jul 17 21:58:01.084951 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.084930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:01.202043 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.202004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.202328 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.202308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.202543 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.202525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.202682 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.202666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xrr\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.202864 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.202849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.203084 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.203063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.203243 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.203220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.203378 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.203276 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.203378 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.203339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.303852 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.303992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304015 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xrr\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304368 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304368 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304289 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304368 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304355 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304531 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304498 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.304700 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.304676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.306735 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.306713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.306842 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.306814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.312039 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.312012 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xrr\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.312628 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.312606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token\") pod \"e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.382912 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.382851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:01.521424 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.521400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:01.527455 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.527424 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:01.527554 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.527490 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:01.527554 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:01.527533 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:02.041227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:02.041188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerStarted","Data":"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea"} Jul 17 21:58:02.041426 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:02.041249 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerStarted","Data":"3e1a08de536347577c8974a0c42766087d48825429133d68e40b4f557feb5d89"} Jul 17 21:58:02.062420 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:02.062372 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podStartSLOduration=1.062358396 podStartE2EDuration="1.062358396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:02.060474 +0000 UTC m=+2955.365472930" watchObservedRunningTime="2026-07-17 21:58:02.062358396 +0000 UTC m=+2955.367357327" Jul 17 21:58:02.383316 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:02.383224 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:02.388354 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:02.388331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:03.046672 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:03.046639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:03.047677 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:03.047659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:04.208408 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.208364 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:04.211985 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.211967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.214669 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.214647 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-380122-serving-cert\"" Jul 17 21:58:04.221844 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.221820 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:04.335869 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.335834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.336046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.335905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7pv\" (UniqueName: \"kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.336046 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.335959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.436580 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.436547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.436806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.436601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7pv\" (UniqueName: \"kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.436806 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.436628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.437310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.437290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.439212 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.439190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.445140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.445118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7pv\" (UniqueName: \"kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv\") pod \"maas-api-e2e-api-a-380122-845695db54-7xr6s\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.525271 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.525245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:04.673076 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.673052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:04.674693 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:58:04.674666 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e7214e_85fc_42c5_ace8_bcb580a6cc84.slice/crio-414c3d24c0d3fe28e06683666e0018da00d891cb979d7d1de723e534ac03543c WatchSource:0}: Error finding container 414c3d24c0d3fe28e06683666e0018da00d891cb979d7d1de723e534ac03543c: Status 404 returned error can't find the container with id 414c3d24c0d3fe28e06683666e0018da00d891cb979d7d1de723e534ac03543c Jul 17 21:58:04.676153 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:04.676136 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:58:05.057459 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:05.057425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" event={"ID":"b0e7214e-85fc-42c5-ace8-bcb580a6cc84","Type":"ContainerStarted","Data":"414c3d24c0d3fe28e06683666e0018da00d891cb979d7d1de723e534ac03543c"} Jul 17 21:58:06.063836 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:06.063804 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" event={"ID":"b0e7214e-85fc-42c5-ace8-bcb580a6cc84","Type":"ContainerStarted","Data":"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b"} Jul 17 21:58:06.064208 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:06.063917 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:06.082565 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:06.082519 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" podStartSLOduration=1.535816947 podStartE2EDuration="2.082504913s" totalImagesPullingTime="546.687966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:06.080404148 +0000 UTC m=+2959.385403090" watchObservedRunningTime="2026-07-17 21:58:06.082504913 +0000 UTC m=+2959.387503843" Jul 17 21:58:12.079891 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:12.079810 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:18.048236 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.048201 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 21:58:18.048642 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.048256 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 21:58:18.128975 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.128941 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerID="0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea" exitCode=0 Jul 17 21:58:18.129145 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.129018 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerDied","Data":"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea"} Jul 17 21:58:18.129451 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.129437 2573 scope.go:117] "RemoveContainer" containerID="0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea" Jul 17 21:58:18.130757 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.130721 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:18.130883 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.130813 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:18.130883 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:18.130847 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:58:19.136310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:19.136275 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerStarted","Data":"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e"} Jul 17 21:58:19.384027 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:19.383984 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:19.384249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:19.384225 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 21:58:19.384329 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:19.384283 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 21:58:20.383653 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:20.383611 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 21:58:20.384063 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:20.383689 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 21:58:21.383546 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:21.383509 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 21:58:21.383717 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:21.383566 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 21:58:22.383933 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:22.383899 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 21:58:22.384322 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:22.383954 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 21:58:23.386798 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:23.386747 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:23.387171 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:23.387048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:23.387881 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:23.387861 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:35.828166 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:35.828122 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:35.828690 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:35.828395 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" podUID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" containerName="maas-api" containerID="cri-o://1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b" gracePeriod=30 Jul 17 21:58:36.078695 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.078634 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:36.154223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.154191 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca\") pod \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " Jul 17 21:58:36.154411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.154235 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls\") pod \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " Jul 17 21:58:36.154411 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.154310 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc7pv\" (UniqueName: \"kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv\") pod \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\" (UID: \"b0e7214e-85fc-42c5-ace8-bcb580a6cc84\") " Jul 17 21:58:36.154609 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.154583 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0e7214e-85fc-42c5-ace8-bcb580a6cc84" (UID: "b0e7214e-85fc-42c5-ace8-bcb580a6cc84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:58:36.154711 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.154696 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:36.156392 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.156365 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0e7214e-85fc-42c5-ace8-bcb580a6cc84" (UID: "b0e7214e-85fc-42c5-ace8-bcb580a6cc84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:58:36.156518 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.156495 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv" (OuterVolumeSpecName: "kube-api-access-sc7pv") pod "b0e7214e-85fc-42c5-ace8-bcb580a6cc84" (UID: "b0e7214e-85fc-42c5-ace8-bcb580a6cc84"). InnerVolumeSpecName "kube-api-access-sc7pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:58:36.226140 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.226114 2573 generic.go:356] "Generic (PLEG): container finished" podID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" containerID="1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b" exitCode=0 Jul 17 21:58:36.226249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.226153 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" event={"ID":"b0e7214e-85fc-42c5-ace8-bcb580a6cc84","Type":"ContainerDied","Data":"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b"} Jul 17 21:58:36.226249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.226171 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" Jul 17 21:58:36.226249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.226223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s" event={"ID":"b0e7214e-85fc-42c5-ace8-bcb580a6cc84","Type":"ContainerDied","Data":"414c3d24c0d3fe28e06683666e0018da00d891cb979d7d1de723e534ac03543c"} Jul 17 21:58:36.226249 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.226247 2573 scope.go:117] "RemoveContainer" containerID="1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b" Jul 17 21:58:36.235314 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.235295 2573 scope.go:117] "RemoveContainer" containerID="1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b" Jul 17 21:58:36.235551 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:58:36.235532 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b\": container with ID starting with 1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b not found: ID does not exist" containerID="1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b" Jul 17 21:58:36.235602 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.235559 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b"} err="failed to get container status \"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b\": rpc error: code = NotFound desc = could not find container \"1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b\": container with ID starting with 1d38233bdfa27b1e81b195027e0c7b2ab2364c71bdc28d07461967f7be5eee5b not found: ID does not exist" Jul 17 21:58:36.247524 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.247503 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:36.251405 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.251376 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380122-845695db54-7xr6s"] Jul 17 21:58:36.255478 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.255453 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:36.255555 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:36.255478 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7pv\" (UniqueName: \"kubernetes.io/projected/b0e7214e-85fc-42c5-ace8-bcb580a6cc84-kube-api-access-sc7pv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:37.237959 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:37.237931 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" path="/var/lib/kubelet/pods/b0e7214e-85fc-42c5-ace8-bcb580a6cc84/volumes" Jul 17 21:58:47.439194 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:47.439155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:58:47.450167 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:47.450150 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 21:58:50.751198 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:50.751163 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:50.751640 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:50.751454 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" containerID="cri-o://9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e" gracePeriod=30 Jul 17 21:58:53.390278 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:53.390231 2573 patch_prober.go:28] interesting pod/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 21:58:53.390741 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:53.390314 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 21:58:56.006026 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.006003 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:56.146444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.146355 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.146444 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.146395 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.147463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.146487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.147463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.147001 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.147463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.147105 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.147463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.147144 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7xrr\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.147463 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.147199 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:58:56.147849 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.147465 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148113 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148161 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148195 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket\") pod \"5a6d23f5-0c32-4646-b5ca-98549e624d30\" (UID: \"5a6d23f5-0c32-4646-b5ca-98549e624d30\") " Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148425 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148837 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-data\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148858 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6d23f5-0c32-4646-b5ca-98549e624d30-istiod-ca-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.149222 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.148875 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.149665 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.149545 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:58:56.149730 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.149658 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:58:56.149896 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.149868 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:58:56.152832 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.150607 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:58:56.152832 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.150915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 21:58:56.152832 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.152292 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr" (OuterVolumeSpecName: "kube-api-access-c7xrr") pod "5a6d23f5-0c32-4646-b5ca-98549e624d30" (UID: "5a6d23f5-0c32-4646-b5ca-98549e624d30"). InnerVolumeSpecName "kube-api-access-c7xrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:58:56.249822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249765 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xrr\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-kube-api-access-c7xrr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.249822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249819 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-podinfo\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.249822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249829 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-workload-socket\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.249822 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249838 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-credential-socket\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.250098 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249847 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-envoy\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.250098 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.249855 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6d23f5-0c32-4646-b5ca-98549e624d30-istio-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 21:58:56.327552 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.327521 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerID="9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e" exitCode=0 Jul 17 21:58:56.327747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.327592 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" Jul 17 21:58:56.327747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.327604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerDied","Data":"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e"} Jul 17 21:58:56.327747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.327659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh" event={"ID":"5a6d23f5-0c32-4646-b5ca-98549e624d30","Type":"ContainerDied","Data":"3e1a08de536347577c8974a0c42766087d48825429133d68e40b4f557feb5d89"} Jul 17 21:58:56.327747 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.327681 2573 scope.go:117] "RemoveContainer" containerID="9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e" Jul 17 21:58:56.337215 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.337199 2573 scope.go:117] "RemoveContainer" containerID="0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea" Jul 17 21:58:56.345150 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.345133 2573 scope.go:117] "RemoveContainer" containerID="9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e" Jul 17 21:58:56.345398 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:58:56.345378 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e\": container with ID starting with 9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e not found: ID does not exist" containerID="9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e" Jul 17 21:58:56.345478 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.345411 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e"} err="failed to get container status \"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e\": rpc error: code = NotFound desc = could not find container \"9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e\": container with ID starting with 9112cb84fc2c2980ea25c2a8bd80fb5c47be062854bc9488803c96956c8f1e2e not found: ID does not exist" Jul 17 21:58:56.345478 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.345434 2573 scope.go:117] "RemoveContainer" containerID="0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea" Jul 17 21:58:56.345668 ip-10-0-129-137 kubenswrapper[2573]: E0717 21:58:56.345648 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea\": container with ID starting with 0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea not found: ID does not exist" containerID="0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea" Jul 17 21:58:56.345705 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.345676 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea"} err="failed to get container status \"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea\": rpc error: code = NotFound desc = could not find container \"0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea\": container with ID starting with 0e94cc29c7ee2cbee78a5ce1b0be056e158dd47a1c156fa90865b359491119ea not found: ID does not exist" Jul 17 21:58:56.349096 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.349076 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:56.353383 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:56.353363 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380122-openshift-default-578cbc6c95-7p7kh"] Jul 17 21:58:57.239849 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:58:57.239820 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" path="/var/lib/kubelet/pods/5a6d23f5-0c32-4646-b5ca-98549e624d30/volumes" Jul 17 21:59:17.717110 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717079 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 21:59:17.717541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717503 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717521 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717541 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717539 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717550 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717593 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" containerName="maas-api" Jul 17 21:59:17.717649 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717602 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" containerName="maas-api" Jul 17 21:59:17.717744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717713 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717728 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6d23f5-0c32-4646-b5ca-98549e624d30" containerName="istio-proxy" Jul 17 21:59:17.717744 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.717735 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e7214e-85fc-42c5-ace8-bcb580a6cc84" containerName="maas-api" Jul 17 21:59:17.720540 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.720521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.724082 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.724063 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ac333d-openshift-default-dockercfg-txwvx\"" Jul 17 21:59:17.732848 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.732821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 21:59:17.848269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848269 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848493 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848592 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848592 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848592 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.848592 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.848584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85sn8\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949461 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949657 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949657 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949657 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949657 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949657 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sn8\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.949969 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.949924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.950227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.950157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.950227 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.950172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.950310 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.950284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.952548 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.952522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.952665 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.952641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.960210 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.960188 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:17.960326 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:17.960276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sn8\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8\") pod \"e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:18.031125 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.031102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:18.170144 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.170115 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 21:59:18.171247 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:59:18.171220 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb1402a_1951_4a9b_a1cd_5ebd34dcaa47.slice/crio-e12b2f98249f41005e28e87c58590a0827ab86263f24896f276de246de77818f WatchSource:0}: Error finding container e12b2f98249f41005e28e87c58590a0827ab86263f24896f276de246de77818f: Status 404 returned error can't find the container with id e12b2f98249f41005e28e87c58590a0827ab86263f24896f276de246de77818f Jul 17 21:59:18.173360 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.173329 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:18.173462 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.173408 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:18.173462 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.173454 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:18.443543 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.443458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerStarted","Data":"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690"} Jul 17 21:59:18.443543 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.443521 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerStarted","Data":"e12b2f98249f41005e28e87c58590a0827ab86263f24896f276de246de77818f"} Jul 17 21:59:18.467663 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:18.467619 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podStartSLOduration=1.4676038820000001 podStartE2EDuration="1.467603882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:18.465738632 +0000 UTC m=+3031.770737578" watchObservedRunningTime="2026-07-17 21:59:18.467603882 +0000 UTC m=+3031.772602813" Jul 17 21:59:19.031720 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:19.031686 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:19.036726 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:19.036703 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:19.448995 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:19.448917 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:19.449741 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:19.449719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:20.458659 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.458621 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 21:59:20.461271 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.461248 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.464132 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.464112 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ac333d-serving-cert\"" Jul 17 21:59:20.474655 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.474631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 21:59:20.574167 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.574127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g2v\" (UniqueName: \"kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.574371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.574271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.574371 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.574361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.675466 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.675425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g2v\" (UniqueName: \"kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.675663 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.675516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.675663 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.675595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.676253 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.676229 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.678168 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.678147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.688223 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.688197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g2v\" (UniqueName: \"kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v\") pod \"maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.774239 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.774213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:20.911011 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:20.910985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 21:59:20.913141 ip-10-0-129-137 kubenswrapper[2573]: W0717 21:59:20.913111 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea72d6f_b6e2_49c5_93cc_5184a5daa72a.slice/crio-b4794c9dfc8cef86089ace73c1fd56b40185670ba5f13284576075598d203c28 WatchSource:0}: Error finding container b4794c9dfc8cef86089ace73c1fd56b40185670ba5f13284576075598d203c28: Status 404 returned error can't find the container with id b4794c9dfc8cef86089ace73c1fd56b40185670ba5f13284576075598d203c28 Jul 17 21:59:21.462239 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:21.462206 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" event={"ID":"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a","Type":"ContainerStarted","Data":"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae"} Jul 17 21:59:21.462699 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:21.462265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" event={"ID":"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a","Type":"ContainerStarted","Data":"b4794c9dfc8cef86089ace73c1fd56b40185670ba5f13284576075598d203c28"} Jul 17 21:59:21.462699 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:21.462461 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:21.482296 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:21.482233 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" podStartSLOduration=1.157342722 podStartE2EDuration="1.482214115s" totalImagesPullingTime="324.871393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:21.480945145 +0000 UTC m=+3034.785944106" watchObservedRunningTime="2026-07-17 21:59:21.482214115 +0000 UTC m=+3034.787213047" Jul 17 21:59:27.472060 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:27.472031 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 21:59:34.545168 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:34.544723 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:46422->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 17 21:59:34.545168 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:34.544843 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:46422->10.134.0.55:15021: read: connection reset by peer" Jul 17 21:59:35.535835 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.535799 2573 generic.go:356] "Generic (PLEG): container finished" podID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerID="68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690" exitCode=0 Jul 17 21:59:35.536014 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.535860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerDied","Data":"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690"} Jul 17 21:59:35.536285 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.536270 2573 scope.go:117] "RemoveContainer" containerID="68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690" Jul 17 21:59:35.537423 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.537399 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:35.537487 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.537472 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:35.537526 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:35.537504 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 21:59:36.542562 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:36.542523 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerStarted","Data":"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd"} Jul 17 21:59:37.032195 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:37.032157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:37.032494 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:37.032467 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 21:59:37.032542 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:37.032522 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 21:59:38.031574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:38.031493 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 21:59:38.031574 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:38.031547 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 21:59:39.032422 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:39.032388 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 21:59:39.032834 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:39.032444 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 21:59:40.031920 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:40.031870 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 21:59:40.032105 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:40.031966 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 21:59:41.034742 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:41.034712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:41.035200 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:41.035001 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 21:59:41.035653 ip-10-0-129-137 kubenswrapper[2573]: I0717 21:59:41.035637 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 22:02:58.118644 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.118567 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:02:58.122469 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.122451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.125754 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.125733 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"kube-root-ca.crt\"" Jul 17 22:02:58.127072 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.127050 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"default-dockercfg-bnws9\"" Jul 17 22:02:58.127072 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.127070 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"openshift-service-ca.crt\"" Jul 17 22:02:58.127237 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.127137 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"sub-test-model-ac333d-kserve-self-signed-certs\"" Jul 17 22:02:58.134118 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.134099 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:02:58.228945 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.228914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.229133 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.228953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.229133 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.229023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgln9\" (UniqueName: \"kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.229133 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.229045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.229133 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.229068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.229309 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.229158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330044 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgln9\" (UniqueName: \"kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330044 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330299 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330299 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330299 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330299 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330524 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330524 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.330605 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.330540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.332592 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.332574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.332741 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.332727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.338559 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.338536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgln9\" (UniqueName: \"kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9\") pod \"sub-test-model-ac333d-kserve-79f7985ccc-78d7v\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.431402 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.431334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:02:58.564511 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.564482 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:02:58.566326 ip-10-0-129-137 kubenswrapper[2573]: W0717 22:02:58.566298 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod409e896a_0c73_4a09_868d_21e5d804512b.slice/crio-ccc1fa70f0bf3305f25bbdee44c10916eca64f91c53be5e1234061a02eb773e6 WatchSource:0}: Error finding container ccc1fa70f0bf3305f25bbdee44c10916eca64f91c53be5e1234061a02eb773e6: Status 404 returned error can't find the container with id ccc1fa70f0bf3305f25bbdee44c10916eca64f91c53be5e1234061a02eb773e6 Jul 17 22:02:58.575703 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:58.575676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" event={"ID":"409e896a-0c73-4a09-868d-21e5d804512b","Type":"ContainerStarted","Data":"ccc1fa70f0bf3305f25bbdee44c10916eca64f91c53be5e1234061a02eb773e6"} Jul 17 22:02:59.582498 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:59.582464 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" event={"ID":"409e896a-0c73-4a09-868d-21e5d804512b","Type":"ContainerStarted","Data":"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18"} Jul 17 22:02:59.599651 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:02:59.599606 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" podStartSLOduration=1.599590289 podStartE2EDuration="1.599590289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:59.598724182 +0000 UTC m=+3252.903723114" watchObservedRunningTime="2026-07-17 22:02:59.599590289 +0000 UTC m=+3252.904589219" Jul 17 22:03:08.432310 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:08.432260 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:03:08.432847 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:08.432332 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:03:08.445066 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:08.445041 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:03:08.640240 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:08.640207 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:03:47.480164 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:47.480138 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 22:03:47.493135 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:03:47.493111 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vb84k_2923af86-87e3-42e0-b039-7a0643df290b/ovn-acl-logging/0.log" Jul 17 22:04:44.165541 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:04:44.165500 2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-137.ec2.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 22:04:54.166230 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:04:54.166188 2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-137.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 17 22:05:04.166688 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:05:04.166631 2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-137.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085828 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085833 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085873 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085828 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085890 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"default-dockercfg-bnws9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085892 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.085917 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085922 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085951 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085951 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085966 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ac333d-openshift-default-dockercfg-txwvx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.085983 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ac333d-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086014 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-dockercfg-526lw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086018 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086027 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086066 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086058 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086082 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086101 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086115 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086125 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086151 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086156 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086171 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086176 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.086568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086154 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086208 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086216 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086226 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086228 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086241 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086262 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mbgfd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086264 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zxvx6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086289 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086292 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086302 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086303 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086323 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086332 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086342 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086343 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086405 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086430 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ptj6w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.087366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086456 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086462 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086463 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086506 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086523 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086529 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086523 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086545 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086552 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086569 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086574 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086584 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086596 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086605 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jfj62\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086611 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086622 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhn5x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086628 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"sub-test-model-ac333d-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086630 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086644 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086650 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086653 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086669 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086677 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086680 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086685 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086689 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086710 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086715 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086724 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fk624usp9g9d2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086722 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086745 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086749 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ldvks\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086755 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kvd2g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086756 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086799 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086806 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.088668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086812 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gls7b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086833 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086841 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086847 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086855 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086871 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-avvcaeankvld7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086887 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086902 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086904 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086903 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086916 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086922 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-csvmh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086920 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086943 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086949 2573 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086959 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086960 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086972 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnjqr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089283 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.086983 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087003 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087017 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087037 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087053 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087060 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087067 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087067 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087081 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087097 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087100 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087099 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xngtz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087121 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087122 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087133 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087144 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087153 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087151 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.089841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087155 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087170 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087175 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087176 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8g3cri9p3iv29\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087187 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087190 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087202 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087207 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087215 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087205 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087234 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087233 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087243 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087243 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087251 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087258 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087259 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087234 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090379 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087276 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087283 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087282 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4ml75\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087300 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087298 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087321 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087335 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:05:10.087332 2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-137.ec2.internal?timeout=10s\": http2: client connection lost" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087361 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087380 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087406 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ntt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087402 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087420 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087432 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087437 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087453 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087458 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087464 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.090939 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087485 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087488 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087502 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087514 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087516 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cqd67\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087521 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2nzg7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087515 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087527 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087490 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087534 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087557 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087534 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087565 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087568 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087543 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087553 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087557 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tz99s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087578 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.091509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087577 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.092184 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087586 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.092184 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.087594 2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 17 22:05:10.889359 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.889325 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 22:05:10.889991 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.889973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\"" Jul 17 22:05:10.892380 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.892361 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 22:05:10.899626 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.899604 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\"" Jul 17 22:05:10.913254 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.913230 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"sub-test-model-ac333d-kserve-self-signed-certs\"" Jul 17 22:05:10.913930 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.913912 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 22:05:10.915038 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.915021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\"" Jul 17 22:05:10.918131 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.918116 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 22:05:10.918205 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.918168 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"default-dockercfg-bnws9\"" Jul 17 22:05:10.920864 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.920846 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 22:05:10.928605 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.928588 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 22:05:10.934851 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.934833 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 22:05:10.946376 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.946347 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 22:05:10.962181 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.962158 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 22:05:10.969701 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.969683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\"" Jul 17 22:05:10.975151 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.975132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhn5x\"" Jul 17 22:05:10.984381 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.984365 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 22:05:10.987142 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:10.987126 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.016850 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.016829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\"" Jul 17 22:05:11.018287 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.018274 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.024980 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.024955 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 22:05:11.028804 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.028782 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 22:05:11.029296 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.029284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kvd2g\"" Jul 17 22:05:11.039587 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.039568 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 22:05:11.041005 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.040990 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 22:05:11.042194 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.042175 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 22:05:11.049580 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.049560 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.054934 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.054917 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tz99s\"" Jul 17 22:05:11.062274 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.062251 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 22:05:11.065374 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.065361 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fk624usp9g9d2\"" Jul 17 22:05:11.072910 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.072892 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 22:05:11.072987 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.072897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 22:05:11.074287 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.074270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ac333d-openshift-default-dockercfg-txwvx\"" Jul 17 22:05:11.079658 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.079643 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\"" Jul 17 22:05:11.080593 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.080579 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 22:05:11.092729 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.092712 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 22:05:11.092729 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.092724 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 22:05:11.094365 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.094350 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 22:05:11.107104 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.107087 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 22:05:11.107912 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.107892 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 22:05:11.113335 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.113316 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 22:05:11.119573 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.119558 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 22:05:11.122369 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.122354 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 22:05:11.127921 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.127902 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 22:05:11.136245 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.136226 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 22:05:11.140333 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.140281 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 22:05:11.141645 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.141617 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.141904 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.141886 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.147185 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.147167 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 22:05:11.148552 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.148533 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ldvks\"" Jul 17 22:05:11.158109 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.158090 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 22:05:11.166841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.166826 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 22:05:11.172402 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.172385 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 22:05:11.178290 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.178270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 22:05:11.179668 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.179654 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 22:05:11.181006 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.180988 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 22:05:11.186365 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.186349 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.186365 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.186359 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 22:05:11.188405 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.188389 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 22:05:11.190435 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.190398 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 22:05:11.195158 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.195139 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.197970 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.197955 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.198236 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.198224 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 22:05:11.199292 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.199279 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zxvx6\"" Jul 17 22:05:11.204561 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.204547 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-csvmh\"" Jul 17 22:05:11.205574 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.205561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.211950 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.211933 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8g3cri9p3iv29\"" Jul 17 22:05:11.220844 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.220822 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 22:05:11.229066 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.229049 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.230475 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.230458 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 22:05:11.234600 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.234582 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 22:05:11.239684 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.239667 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.246102 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.246085 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 22:05:11.248594 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.248578 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 22:05:11.251497 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.251474 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gls7b\"" Jul 17 22:05:11.252676 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.252660 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 22:05:11.256478 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.256461 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 22:05:11.260477 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.260458 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 22:05:11.261607 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.261591 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\"" Jul 17 22:05:11.263564 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.263545 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:05:11.272416 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.272396 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 22:05:11.275203 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.275184 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.278354 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.278337 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.298695 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.298667 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 22:05:11.298936 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.298718 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.301512 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.301498 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.306229 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.306210 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 22:05:11.309655 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.309614 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ac333d-serving-cert\"" Jul 17 22:05:11.310580 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.310562 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.318612 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.318589 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 22:05:11.322009 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.321987 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 22:05:11.323171 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.323153 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.328499 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.328482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.341351 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.341331 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\"" Jul 17 22:05:11.341455 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.341422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 22:05:11.346592 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.346563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 22:05:11.350758 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.350738 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\"" Jul 17 22:05:11.355649 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.355625 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\"" Jul 17 22:05:11.356913 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.356900 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ptj6w\"" Jul 17 22:05:11.360937 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.360920 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cqd67\"" Jul 17 22:05:11.365105 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.365087 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 22:05:11.369742 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.369723 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-526lw\"" Jul 17 22:05:11.371515 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.371500 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 22:05:11.375685 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.375670 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jfj62\"" Jul 17 22:05:11.378068 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.378052 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\"" Jul 17 22:05:11.382400 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.382384 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mbgfd\"" Jul 17 22:05:11.383420 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.383404 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.390721 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.390672 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnjqr\"" Jul 17 22:05:11.390926 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.390911 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.393566 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.393550 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 22:05:11.403912 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.403894 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.404938 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.404923 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.407916 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.407900 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 22:05:11.413110 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.413086 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 22:05:11.425991 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.425968 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.431561 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.431544 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 22:05:11.438721 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.438703 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 22:05:11.455085 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.455067 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.460565 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.460546 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4ml75\"" Jul 17 22:05:11.460695 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.460563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\"" Jul 17 22:05:11.464840 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.464825 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 22:05:11.472307 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.472282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xngtz\"" Jul 17 22:05:11.479509 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.479490 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.479598 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.479567 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 22:05:11.483736 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.483719 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\"" Jul 17 22:05:11.485014 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.485000 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.497677 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.497656 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 22:05:11.498976 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.498958 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 22:05:11.502020 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.502002 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 17 22:05:11.514805 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.514786 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 22:05:11.530350 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.530333 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.549990 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.549973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\"" Jul 17 22:05:11.570300 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.570278 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 22:05:11.590188 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.590169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 22:05:11.609590 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.609564 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65ntt\"" Jul 17 22:05:11.629334 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.629313 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 22:05:11.649044 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.648977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-avvcaeankvld7\"" Jul 17 22:05:11.669677 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.669655 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\"" Jul 17 22:05:11.689609 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.689588 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 22:05:11.709691 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.709666 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 22:05:11.729184 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.729159 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 22:05:11.749387 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.749363 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.770458 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.770435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\"" Jul 17 22:05:11.789355 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.789332 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 22:05:11.810176 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.810155 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 22:05:11.829156 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.829136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 22:05:11.849255 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.849236 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 22:05:11.870898 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.870877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 22:05:11.889404 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.889381 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 22:05:11.909197 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.909142 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 22:05:11.929342 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.929324 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 22:05:11.950052 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.950034 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 22:05:11.969436 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.969414 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 22:05:11.989788 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:11.989755 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2nzg7\"" Jul 17 22:05:12.009179 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.009156 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 22:05:12.029733 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.029714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 22:05:12.050120 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.050100 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 22:05:12.070021 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.069996 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 22:05:12.096199 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.096179 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 22:05:12.110116 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.110092 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 22:05:12.130764 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.130747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 22:05:12.150157 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.150135 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\"" Jul 17 22:05:12.170386 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.170325 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 22:05:12.190306 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.190285 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 22:05:12.209679 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.209651 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 22:05:12.230069 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.230045 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\"" Jul 17 22:05:12.249817 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.249793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 22:05:12.269147 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.269125 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 22:05:12.289115 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.289097 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 22:05:12.310030 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.310007 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 22:05:12.329479 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:12.329458 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 22:05:26.284887 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.284840 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" podUID="409e896a-0c73-4a09-868d-21e5d804512b" containerName="main" containerID="cri-o://3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18" gracePeriod=45 Jul 17 22:05:26.551436 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.551411 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:05:26.697999 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.697965 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698193 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698017 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698193 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698060 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698193 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698117 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgln9\" (UniqueName: \"kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698193 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698152 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698445 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698219 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache\") pod \"409e896a-0c73-4a09-868d-21e5d804512b\" (UID: \"409e896a-0c73-4a09-868d-21e5d804512b\") " Jul 17 22:05:26.698445 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698252 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:05:26.698445 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698344 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home" (OuterVolumeSpecName: "home") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:05:26.698607 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698513 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache" (OuterVolumeSpecName: "model-cache") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:05:26.698607 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698590 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:26.698684 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.698610 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:26.700537 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.700514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm" (OuterVolumeSpecName: "dshm") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:05:26.700626 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.700557 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:05:26.700662 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.700634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9" (OuterVolumeSpecName: "kube-api-access-xgln9") pod "409e896a-0c73-4a09-868d-21e5d804512b" (UID: "409e896a-0c73-4a09-868d-21e5d804512b"). InnerVolumeSpecName "kube-api-access-xgln9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:05:26.799568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.799483 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:26.799568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.799512 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgln9\" (UniqueName: \"kubernetes.io/projected/409e896a-0c73-4a09-868d-21e5d804512b-kube-api-access-xgln9\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:26.799568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.799524 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409e896a-0c73-4a09-868d-21e5d804512b-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:26.799568 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:26.799533 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409e896a-0c73-4a09-868d-21e5d804512b-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:05:27.326132 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.326102 2573 generic.go:356] "Generic (PLEG): container finished" podID="409e896a-0c73-4a09-868d-21e5d804512b" containerID="3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18" exitCode=0 Jul 17 22:05:27.326577 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.326146 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" Jul 17 22:05:27.326577 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.326150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" event={"ID":"409e896a-0c73-4a09-868d-21e5d804512b","Type":"ContainerDied","Data":"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18"} Jul 17 22:05:27.326577 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.326292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v" event={"ID":"409e896a-0c73-4a09-868d-21e5d804512b","Type":"ContainerDied","Data":"ccc1fa70f0bf3305f25bbdee44c10916eca64f91c53be5e1234061a02eb773e6"} Jul 17 22:05:27.326577 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.326329 2573 scope.go:117] "RemoveContainer" containerID="3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18" Jul 17 22:05:27.335108 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.335091 2573 scope.go:117] "RemoveContainer" containerID="3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18" Jul 17 22:05:27.335352 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:05:27.335335 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18\": container with ID starting with 3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18 not found: ID does not exist" containerID="3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18" Jul 17 22:05:27.335413 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.335360 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18"} err="failed to get container status \"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18\": rpc error: code = NotFound desc = could not find container \"3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18\": container with ID starting with 3cf766ad444520f72a9a946134d4f05a9dddca19ec4ae6999a30fb956c51aa18 not found: ID does not exist" Jul 17 22:05:27.342412 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.342391 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:05:27.346130 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:27.346109 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/sub-test-model-ac333d-kserve-79f7985ccc-78d7v"] Jul 17 22:05:29.238978 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:05:29.238945 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409e896a-0c73-4a09-868d-21e5d804512b" path="/var/lib/kubelet/pods/409e896a-0c73-4a09-868d-21e5d804512b/volumes" Jul 17 22:06:31.461848 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.461816 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158998704"] Jul 17 22:06:31.462272 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.462224 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409e896a-0c73-4a09-868d-21e5d804512b" containerName="main" Jul 17 22:06:31.462272 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.462235 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="409e896a-0c73-4a09-868d-21e5d804512b" containerName="main" Jul 17 22:06:31.462356 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.462344 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="409e896a-0c73-4a09-868d-21e5d804512b" containerName="main" Jul 17 22:06:31.465219 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.465204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:31.467522 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.467471 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rxnlj\"" Jul 17 22:06:31.469576 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.469550 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158998704"] Jul 17 22:06:31.537153 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.537122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxvw\" (UniqueName: \"kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw\") pod \"test-curl-7257-139628158998704\" (UID: \"75b62400-940a-487b-a13a-4e54ab565174\") " pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:31.637591 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.637561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxvw\" (UniqueName: \"kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw\") pod \"test-curl-7257-139628158998704\" (UID: \"75b62400-940a-487b-a13a-4e54ab565174\") " pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:31.645905 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.645877 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxvw\" (UniqueName: \"kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw\") pod \"test-curl-7257-139628158998704\" (UID: \"75b62400-940a-487b-a13a-4e54ab565174\") " pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:31.776076 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.776039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:31.910054 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.910025 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158998704"] Jul 17 22:06:31.910474 ip-10-0-129-137 kubenswrapper[2573]: W0717 22:06:31.910450 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75b62400_940a_487b_a13a_4e54ab565174.slice/crio-bef171d506212f1380ae4fa02a91c9049036b46d8481cf2acf4b3840ce138a95 WatchSource:0}: Error finding container bef171d506212f1380ae4fa02a91c9049036b46d8481cf2acf4b3840ce138a95: Status 404 returned error can't find the container with id bef171d506212f1380ae4fa02a91c9049036b46d8481cf2acf4b3840ce138a95 Jul 17 22:06:31.912365 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:31.912346 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:06:32.657434 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:32.657410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158998704" event={"ID":"75b62400-940a-487b-a13a-4e54ab565174","Type":"ContainerStarted","Data":"bef171d506212f1380ae4fa02a91c9049036b46d8481cf2acf4b3840ce138a95"} Jul 17 22:06:33.663983 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:33.663950 2573 generic.go:356] "Generic (PLEG): container finished" podID="75b62400-940a-487b-a13a-4e54ab565174" containerID="34acb0ce597e1fc6660cf3aed6d27844e7577c6ddc868801f6221ecda615114d" exitCode=0 Jul 17 22:06:33.664355 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:33.664009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158998704" event={"ID":"75b62400-940a-487b-a13a-4e54ab565174","Type":"ContainerDied","Data":"34acb0ce597e1fc6660cf3aed6d27844e7577c6ddc868801f6221ecda615114d"} Jul 17 22:06:34.802737 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.802713 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:34.834605 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.834581 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158998704_75b62400-940a-487b-a13a-4e54ab565174/test-curl-7257-139628158998704/0.log" Jul 17 22:06:34.860162 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.860107 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158998704"] Jul 17 22:06:34.863104 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.863086 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158998704"] Jul 17 22:06:34.967619 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.967542 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zxvw\" (UniqueName: \"kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw\") pod \"75b62400-940a-487b-a13a-4e54ab565174\" (UID: \"75b62400-940a-487b-a13a-4e54ab565174\") " Jul 17 22:06:34.969679 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:34.969649 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw" (OuterVolumeSpecName: "kube-api-access-9zxvw") pod "75b62400-940a-487b-a13a-4e54ab565174" (UID: "75b62400-940a-487b-a13a-4e54ab565174"). InnerVolumeSpecName "kube-api-access-9zxvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:06:35.069161 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:35.069131 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxvw\" (UniqueName: \"kubernetes.io/projected/75b62400-940a-487b-a13a-4e54ab565174-kube-api-access-9zxvw\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:35.239141 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:35.239062 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b62400-940a-487b-a13a-4e54ab565174" path="/var/lib/kubelet/pods/75b62400-940a-487b-a13a-4e54ab565174/volumes" Jul 17 22:06:35.674566 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:35.674532 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158998704" Jul 17 22:06:35.674749 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:35.674579 2573 scope.go:117] "RemoveContainer" containerID="34acb0ce597e1fc6660cf3aed6d27844e7577c6ddc868801f6221ecda615114d" Jul 17 22:06:37.483026 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.482994 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158805312"] Jul 17 22:06:37.483622 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.483602 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b62400-940a-487b-a13a-4e54ab565174" containerName="test-curl-7257-139628158998704" Jul 17 22:06:37.483714 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.483625 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b62400-940a-487b-a13a-4e54ab565174" containerName="test-curl-7257-139628158998704" Jul 17 22:06:37.483794 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.483763 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b62400-940a-487b-a13a-4e54ab565174" containerName="test-curl-7257-139628158998704" Jul 17 22:06:37.488354 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.488326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:37.490386 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.490365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"] Jul 17 22:06:37.491195 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.491178 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rxnlj\"" Jul 17 22:06:37.593417 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.593378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6zh\" (UniqueName: \"kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh\") pod \"test-curl-7257-139628158805312\" (UID: \"64068b9a-2ed5-4209-81ea-d83f37cc808f\") " pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:37.695087 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.695052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6zh\" (UniqueName: \"kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh\") pod \"test-curl-7257-139628158805312\" (UID: \"64068b9a-2ed5-4209-81ea-d83f37cc808f\") " pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:37.703666 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.703634 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6zh\" (UniqueName: \"kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh\") pod \"test-curl-7257-139628158805312\" (UID: \"64068b9a-2ed5-4209-81ea-d83f37cc808f\") " pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:37.798841 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.798811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:37.931516 ip-10-0-129-137 kubenswrapper[2573]: W0717 22:06:37.931485 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64068b9a_2ed5_4209_81ea_d83f37cc808f.slice/crio-4cbadc20c7b4ae6b31cfb38b314e739d63bda4ea1589cd5d3564a7d643aac702 WatchSource:0}: Error finding container 4cbadc20c7b4ae6b31cfb38b314e739d63bda4ea1589cd5d3564a7d643aac702: Status 404 returned error can't find the container with id 4cbadc20c7b4ae6b31cfb38b314e739d63bda4ea1589cd5d3564a7d643aac702 Jul 17 22:06:37.931705 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:37.931683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"] Jul 17 22:06:38.695267 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:38.695231 2573 generic.go:356] "Generic (PLEG): container finished" podID="64068b9a-2ed5-4209-81ea-d83f37cc808f" containerID="7483b1a8938d6dfb5b0f3d6cf24bd41787ea27da702649badc6a7a21e28fc7e3" exitCode=0 Jul 17 22:06:38.695683 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:38.695315 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158805312" event={"ID":"64068b9a-2ed5-4209-81ea-d83f37cc808f","Type":"ContainerDied","Data":"7483b1a8938d6dfb5b0f3d6cf24bd41787ea27da702649badc6a7a21e28fc7e3"} Jul 17 22:06:38.695683 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:38.695359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158805312" event={"ID":"64068b9a-2ed5-4209-81ea-d83f37cc808f","Type":"ContainerStarted","Data":"4cbadc20c7b4ae6b31cfb38b314e739d63bda4ea1589cd5d3564a7d643aac702"} Jul 17 22:06:39.833580 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.833554 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:39.852760 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.852736 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158805312_64068b9a-2ed5-4209-81ea-d83f37cc808f/test-curl-7257-139628158805312/0.log" Jul 17 22:06:39.873511 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.873486 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"] Jul 17 22:06:39.878950 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.878923 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"] Jul 17 22:06:39.997395 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.997359 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158806320"] Jul 17 22:06:39.997903 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.997884 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64068b9a-2ed5-4209-81ea-d83f37cc808f" containerName="test-curl-7257-139628158805312" Jul 17 22:06:39.998013 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.997904 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64068b9a-2ed5-4209-81ea-d83f37cc808f" containerName="test-curl-7257-139628158805312" Jul 17 22:06:39.998067 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:39.998022 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64068b9a-2ed5-4209-81ea-d83f37cc808f" containerName="test-curl-7257-139628158805312" Jul 17 22:06:40.001235 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.001215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:40.005973 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.005953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"] Jul 17 22:06:40.019318 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.019298 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs6zh\" (UniqueName: \"kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh\") pod \"64068b9a-2ed5-4209-81ea-d83f37cc808f\" (UID: \"64068b9a-2ed5-4209-81ea-d83f37cc808f\") " Jul 17 22:06:40.021456 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.021434 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh" (OuterVolumeSpecName: "kube-api-access-gs6zh") pod "64068b9a-2ed5-4209-81ea-d83f37cc808f" (UID: "64068b9a-2ed5-4209-81ea-d83f37cc808f"). InnerVolumeSpecName "kube-api-access-gs6zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:06:40.120973 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.120936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26pq\" (UniqueName: \"kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq\") pod \"test-curl-7257-139628158806320\" (UID: \"8a46ced7-015a-4f97-b8a8-a3b8c10206e7\") " pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:40.121118 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.121100 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6zh\" (UniqueName: \"kubernetes.io/projected/64068b9a-2ed5-4209-81ea-d83f37cc808f-kube-api-access-gs6zh\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:40.221974 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.221943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26pq\" (UniqueName: \"kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq\") pod \"test-curl-7257-139628158806320\" (UID: \"8a46ced7-015a-4f97-b8a8-a3b8c10206e7\") " pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:40.230262 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.230234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26pq\" (UniqueName: \"kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq\") pod \"test-curl-7257-139628158806320\" (UID: \"8a46ced7-015a-4f97-b8a8-a3b8c10206e7\") " pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:40.312035 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.311959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:40.446073 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.446041 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"] Jul 17 22:06:40.447542 ip-10-0-129-137 kubenswrapper[2573]: W0717 22:06:40.447511 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a46ced7_015a_4f97_b8a8_a3b8c10206e7.slice/crio-6c77d085ea7da5efd0b7f9e8f0bc32a7f33d0558eea668b7a8324e69e7a534c4 WatchSource:0}: Error finding container 6c77d085ea7da5efd0b7f9e8f0bc32a7f33d0558eea668b7a8324e69e7a534c4: Status 404 returned error can't find the container with id 6c77d085ea7da5efd0b7f9e8f0bc32a7f33d0558eea668b7a8324e69e7a534c4 Jul 17 22:06:40.707485 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.707391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158806320" event={"ID":"8a46ced7-015a-4f97-b8a8-a3b8c10206e7","Type":"ContainerStarted","Data":"6c77d085ea7da5efd0b7f9e8f0bc32a7f33d0558eea668b7a8324e69e7a534c4"} Jul 17 22:06:40.708608 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.708583 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:40.708722 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:40.708583 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbadc20c7b4ae6b31cfb38b314e739d63bda4ea1589cd5d3564a7d643aac702" Jul 17 22:06:40.718520 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:06:40.718493 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7257-139628158805312\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="64068b9a-2ed5-4209-81ea-d83f37cc808f" pod="opendatahub/test-curl-7257-139628158805312" Jul 17 22:06:41.239284 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:41.239253 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64068b9a-2ed5-4209-81ea-d83f37cc808f" path="/var/lib/kubelet/pods/64068b9a-2ed5-4209-81ea-d83f37cc808f/volumes" Jul 17 22:06:41.715966 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:41.715930 2573 generic.go:356] "Generic (PLEG): container finished" podID="8a46ced7-015a-4f97-b8a8-a3b8c10206e7" containerID="18682b574601f36d057b5a333f219f2e496d192bee7d15eb0cbd2987db2edcad" exitCode=0 Jul 17 22:06:41.716149 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:41.716017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158806320" event={"ID":"8a46ced7-015a-4f97-b8a8-a3b8c10206e7","Type":"ContainerDied","Data":"18682b574601f36d057b5a333f219f2e496d192bee7d15eb0cbd2987db2edcad"} Jul 17 22:06:42.865423 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.865400 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:42.885799 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.885753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158806320_8a46ced7-015a-4f97-b8a8-a3b8c10206e7/test-curl-7257-139628158806320/0.log" Jul 17 22:06:42.910500 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.910471 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"] Jul 17 22:06:42.915956 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.915929 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"] Jul 17 22:06:42.947380 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.947358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w26pq\" (UniqueName: \"kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq\") pod \"8a46ced7-015a-4f97-b8a8-a3b8c10206e7\" (UID: \"8a46ced7-015a-4f97-b8a8-a3b8c10206e7\") " Jul 17 22:06:42.949438 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:42.949415 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq" (OuterVolumeSpecName: "kube-api-access-w26pq") pod "8a46ced7-015a-4f97-b8a8-a3b8c10206e7" (UID: "8a46ced7-015a-4f97-b8a8-a3b8c10206e7"). InnerVolumeSpecName "kube-api-access-w26pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:06:43.048361 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:43.048325 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w26pq\" (UniqueName: \"kubernetes.io/projected/8a46ced7-015a-4f97-b8a8-a3b8c10206e7-kube-api-access-w26pq\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:43.239658 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:43.239624 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a46ced7-015a-4f97-b8a8-a3b8c10206e7" path="/var/lib/kubelet/pods/8a46ced7-015a-4f97-b8a8-a3b8c10206e7/volumes" Jul 17 22:06:43.727840 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:43.727803 2573 scope.go:117] "RemoveContainer" containerID="18682b574601f36d057b5a333f219f2e496d192bee7d15eb0cbd2987db2edcad" Jul 17 22:06:43.727840 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:43.727834 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320" Jul 17 22:06:56.180133 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.180096 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 22:06:56.180593 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.180377 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" podUID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" containerName="maas-api" containerID="cri-o://642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae" gracePeriod=30 Jul 17 22:06:56.450181 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.450160 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 22:06:56.580188 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.580156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls\") pod \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " Jul 17 22:06:56.580355 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.580224 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca\") pod \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " Jul 17 22:06:56.580355 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.580258 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5g2v\" (UniqueName: \"kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v\") pod \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\" (UID: \"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a\") " Jul 17 22:06:56.580606 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.580573 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" (UID: "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:06:56.582315 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.582292 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" (UID: "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:06:56.582543 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.582521 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v" (OuterVolumeSpecName: "kube-api-access-k5g2v") pod "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" (UID: "8ea72d6f-b6e2-49c5-93cc-5184a5daa72a"). InnerVolumeSpecName "kube-api-access-k5g2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:06:56.681813 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.681785 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-maas-api-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:56.681813 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.681809 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-openshift-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:56.681813 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.681819 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5g2v\" (UniqueName: \"kubernetes.io/projected/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a-kube-api-access-k5g2v\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:06:56.802159 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.802120 2573 generic.go:356] "Generic (PLEG): container finished" podID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" containerID="642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae" exitCode=0 Jul 17 22:06:56.802349 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.802175 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" Jul 17 22:06:56.802349 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.802213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" event={"ID":"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a","Type":"ContainerDied","Data":"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae"} Jul 17 22:06:56.802349 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.802289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj" event={"ID":"8ea72d6f-b6e2-49c5-93cc-5184a5daa72a","Type":"ContainerDied","Data":"b4794c9dfc8cef86089ace73c1fd56b40185670ba5f13284576075598d203c28"} Jul 17 22:06:56.802349 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.802310 2573 scope.go:117] "RemoveContainer" containerID="642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae" Jul 17 22:06:56.811687 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.811672 2573 scope.go:117] "RemoveContainer" containerID="642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae" Jul 17 22:06:56.811946 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:06:56.811928 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae\": container with ID starting with 642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae not found: ID does not exist" containerID="642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae" Jul 17 22:06:56.812024 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.811953 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae"} err="failed to get container status \"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae\": rpc error: code = NotFound desc = could not find container \"642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae\": container with ID starting with 642730fa1fbadd189c737629ca6c8bb8e4f9bf1264619b952cd0c5a0d8bc76ae not found: ID does not exist" Jul 17 22:06:56.823363 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.823318 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 22:06:56.826694 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:56.826673 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ac333d-7b947ff5d-xqxvj"] Jul 17 22:06:57.238333 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:06:57.238247 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" path="/var/lib/kubelet/pods/8ea72d6f-b6e2-49c5-93cc-5184a5daa72a/volumes" Jul 17 22:07:11.357614 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:11.357530 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 22:07:11.358082 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:11.357731 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" containerID="cri-o://e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd" gracePeriod=30 Jul 17 22:07:16.621603 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.621535 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 22:07:16.780305 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780273 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85sn8\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780508 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780323 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780508 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780349 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780508 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780384 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780508 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780486 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780734 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780532 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780734 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780642 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780734 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780674 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token\") pod \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\" (UID: \"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47\") " Jul 17 22:07:16.780929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780670 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:16.780929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780810 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:07:16.780929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:16.780929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.780691 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:16.781178 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.781166 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istiod-ca-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.781240 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.781186 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.781339 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.781265 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-credential-socket\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.781405 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.781376 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-workload-socket\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.781811 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.781757 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:16.782914 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.782887 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:16.782914 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.782894 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8" (OuterVolumeSpecName: "kube-api-access-85sn8") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "kube-api-access-85sn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:07:16.783051 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.782891 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:07:16.783051 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.783020 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" (UID: "0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:07:16.882138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.882053 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-data\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.882138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.882081 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-envoy\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.882138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.882090 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.882138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.882098 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85sn8\" (UniqueName: \"kubernetes.io/projected/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-kube-api-access-85sn8\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.882138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.882107 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47-istio-podinfo\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 17 22:07:16.907607 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.907574 2573 generic.go:356] "Generic (PLEG): container finished" podID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerID="e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd" exitCode=0 Jul 17 22:07:16.907744 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.907624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" Jul 17 22:07:16.907744 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.907671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerDied","Data":"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd"} Jul 17 22:07:16.907744 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.907732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m" event={"ID":"0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47","Type":"ContainerDied","Data":"e12b2f98249f41005e28e87c58590a0827ab86263f24896f276de246de77818f"} Jul 17 22:07:16.908004 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.907764 2573 scope.go:117] "RemoveContainer" containerID="e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd" Jul 17 22:07:16.917165 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.917148 2573 scope.go:117] "RemoveContainer" containerID="68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690" Jul 17 22:07:16.925102 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.925084 2573 scope.go:117] "RemoveContainer" containerID="e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd" Jul 17 22:07:16.925368 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:07:16.925348 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd\": container with ID starting with e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd not found: ID does not exist" containerID="e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd" Jul 17 22:07:16.925441 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.925380 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd"} err="failed to get container status \"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd\": rpc error: code = NotFound desc = could not find container \"e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd\": container with ID starting with e7c3cda6283a55d42b2adf07240fdc5da3566d786f809e4111049d93cd376efd not found: ID does not exist" Jul 17 22:07:16.925441 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.925403 2573 scope.go:117] "RemoveContainer" containerID="68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690" Jul 17 22:07:16.925642 ip-10-0-129-137 kubenswrapper[2573]: E0717 22:07:16.925624 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690\": container with ID starting with 68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690 not found: ID does not exist" containerID="68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690" Jul 17 22:07:16.925699 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.925650 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690"} err="failed to get container status \"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690\": rpc error: code = NotFound desc = could not find container \"68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690\": container with ID starting with 68f57fbb01c429a0d52565ebea21dd098aff0302c139b858cb2953912a88d690 not found: ID does not exist" Jul 17 22:07:16.928696 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.928676 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 22:07:16.932669 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:16.932646 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ac333d-openshift-default-77d585dd7-4kz2m"] Jul 17 22:07:17.239194 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:17.239119 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" path="/var/lib/kubelet/pods/0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47/volumes" Jul 17 22:07:43.448631 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:43.448597 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-557f54c68b-8gszz_4c4644f6-c066-4767-9727-07223eb066f5/manager/0.log" Jul 17 22:07:44.083501 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:44.083471 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-vjsms_3fec7d27-e113-49aa-bf14-2767ee7782cf/copy-manifests/0.log" Jul 17 22:07:44.091835 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:44.091805 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-vjsms_3fec7d27-e113-49aa-bf14-2767ee7782cf/manager/0.log" Jul 17 22:07:44.426967 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:44.426885 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f68cd45b9-4hwjk_20481527-9c54-4ccd-946e-4b79c0040d2f/maas-api/0.log" Jul 17 22:07:45.816879 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:45.816841 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kzkgl_6b5f708a-4f3a-48af-8c29-a69db4d71e63/postgres/0.log" Jul 17 22:07:48.431151 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:48.431114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-dw7vh_c57535d6-10b6-4d40-a748-f297653e7d91/kube-auth-proxy/0.log" Jul 17 22:07:48.655366 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:48.655328 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/1.log" Jul 17 22:07:48.767170 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:48.767137 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-86qlf_1fafc772-0bfc-433d-9462-e5922835c535/payload-pre-processing/0.log" Jul 17 22:07:48.876746 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:48.876715 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-72zq9_14411f37-d3fb-4c59-be8d-6662499ca5a9/payload-processing/0.log" Jul 17 22:07:49.325856 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:49.325829 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vb5gv_e532825c-cf85-4dc0-804e-bdf30c63ce8e/main/0.log" Jul 17 22:07:49.540963 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:49.540934 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5rwn7_d9067147-c980-4af9-ad04-f4f543d26dd6/main/0.log" Jul 17 22:07:49.654364 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:49.654271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnqwtg_d099393b-490d-4d98-ac6d-56197b0fc90b/main/0.log" Jul 17 22:07:57.556802 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:57.556746 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k2hd6_92967c53-9967-4227-84fa-408e172b8a27/global-pull-secret-syncer/0.log" Jul 17 22:07:57.678787 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:57.678745 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r8m8k_f3173e73-266f-4600-a98b-97d2c3a33af7/konnectivity-agent/0.log" Jul 17 22:07:57.722398 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:07:57.722363 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-137.ec2.internal_f449d6f92faa6a51a14c54944141985e/haproxy/0.log" Jul 17 22:08:05.508612 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.508574 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/alertmanager/0.log" Jul 17 22:08:05.528404 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.528379 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/config-reloader/0.log" Jul 17 22:08:05.552449 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.552427 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/kube-rbac-proxy-web/0.log" Jul 17 22:08:05.572316 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.572295 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/kube-rbac-proxy/0.log" Jul 17 22:08:05.595522 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.595502 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/kube-rbac-proxy-metric/0.log" Jul 17 22:08:05.617347 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.617321 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/prom-label-proxy/0.log" Jul 17 22:08:05.635134 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.635118 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba79ee40-7173-4759-bbd8-6d6a1113b2bb/init-config-reloader/0.log" Jul 17 22:08:05.707799 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.707748 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvd5z_55f622da-ecd7-4413-8629-8cce3056b0ad/kube-state-metrics/0.log" Jul 17 22:08:05.727810 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.727789 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvd5z_55f622da-ecd7-4413-8629-8cce3056b0ad/kube-rbac-proxy-main/0.log" Jul 17 22:08:05.754503 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.754485 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvd5z_55f622da-ecd7-4413-8629-8cce3056b0ad/kube-rbac-proxy-self/0.log" Jul 17 22:08:05.781576 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.781495 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f5c5df4cf-nw5k9_e26bef88-f99c-4dc0-a53e-ab6f4c22040a/metrics-server/0.log" Jul 17 22:08:05.811703 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.811676 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-j462v_164470f9-84d6-41b4-b1be-b867b5f32603/monitoring-plugin/0.log" Jul 17 22:08:05.922230 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.922206 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qhm8_9111c994-45fe-4cf1-ac20-3f2562da51c6/node-exporter/0.log" Jul 17 22:08:05.941090 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.941064 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qhm8_9111c994-45fe-4cf1-ac20-3f2562da51c6/kube-rbac-proxy/0.log" Jul 17 22:08:05.960338 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:05.960318 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qhm8_9111c994-45fe-4cf1-ac20-3f2562da51c6/init-textfile/0.log" Jul 17 22:08:06.061122 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.061034 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-2fvbr_8eb1e02e-a084-43d6-be49-889cc71b330c/kube-rbac-proxy-main/0.log" Jul 17 22:08:06.080363 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.080329 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-2fvbr_8eb1e02e-a084-43d6-be49-889cc71b330c/kube-rbac-proxy-self/0.log" Jul 17 22:08:06.100123 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.100086 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-2fvbr_8eb1e02e-a084-43d6-be49-889cc71b330c/openshift-state-metrics/0.log" Jul 17 22:08:06.134381 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.134351 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/prometheus/0.log" Jul 17 22:08:06.153575 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.153554 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/config-reloader/0.log" Jul 17 22:08:06.172992 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.172971 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/thanos-sidecar/0.log" Jul 17 22:08:06.195365 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.195337 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/kube-rbac-proxy-web/0.log" Jul 17 22:08:06.214008 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.213988 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/kube-rbac-proxy/0.log" Jul 17 22:08:06.232964 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.232934 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/kube-rbac-proxy-thanos/0.log" Jul 17 22:08:06.254927 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.254906 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2662cc6-0b16-46cb-8bb6-3d36ff6f9b38/init-config-reloader/0.log" Jul 17 22:08:06.284899 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.284872 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-225wh_41790925-1420-4927-b9a8-dd8248642037/prometheus-operator/0.log" Jul 17 22:08:06.305239 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.305224 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-225wh_41790925-1420-4927-b9a8-dd8248642037/kube-rbac-proxy/0.log" Jul 17 22:08:06.367136 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.367059 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56555c9968-24q7c_07fa8142-e2c2-4d8d-8257-2d9899ac2418/telemeter-client/0.log" Jul 17 22:08:06.385177 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.385152 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56555c9968-24q7c_07fa8142-e2c2-4d8d-8257-2d9899ac2418/reload/0.log" Jul 17 22:08:06.401835 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.401800 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56555c9968-24q7c_07fa8142-e2c2-4d8d-8257-2d9899ac2418/kube-rbac-proxy/0.log" Jul 17 22:08:06.431330 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.431302 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/thanos-query/0.log" Jul 17 22:08:06.449573 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.449540 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/kube-rbac-proxy-web/0.log" Jul 17 22:08:06.467586 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.467566 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/kube-rbac-proxy/0.log" Jul 17 22:08:06.485428 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.485404 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/prom-label-proxy/0.log" Jul 17 22:08:06.505881 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.505837 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/kube-rbac-proxy-rules/0.log" Jul 17 22:08:06.524958 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:06.524938 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9cdb648f-kd4hg_1df7c130-a6bc-4f63-a54b-9bfe4f41ae2e/kube-rbac-proxy-metrics/0.log" Jul 17 22:08:08.095887 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.095858 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk"] Jul 17 22:08:08.096268 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096256 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" containerName="maas-api" Jul 17 22:08:08.096268 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096269 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" containerName="maas-api" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096289 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a46ced7-015a-4f97-b8a8-a3b8c10206e7" containerName="test-curl-7257-139628158806320" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096295 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a46ced7-015a-4f97-b8a8-a3b8c10206e7" containerName="test-curl-7257-139628158806320" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096308 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096313 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096322 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" Jul 17 22:08:08.096343 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096327 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" Jul 17 22:08:08.096525 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096384 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb1402a-1951-4a9b-a1cd-5ebd34dcaa47" containerName="istio-proxy" Jul 17 22:08:08.096525 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096394 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a46ced7-015a-4f97-b8a8-a3b8c10206e7" containerName="test-curl-7257-139628158806320" Jul 17 22:08:08.096525 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.096402 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea72d6f-b6e2-49c5-93cc-5184a5daa72a" containerName="maas-api" Jul 17 22:08:08.099400 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.099385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.101820 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.101792 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"openshift-service-ca.crt\"" Jul 17 22:08:08.101820 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.101812 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"kube-root-ca.crt\"" Jul 17 22:08:08.102213 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.102192 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4qhz6\"/\"default-dockercfg-tq2z6\"" Jul 17 22:08:08.109168 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.109147 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk"] Jul 17 22:08:08.177557 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.177512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-podres\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.177738 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.177566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2g42\" (UniqueName: \"kubernetes.io/projected/792bb765-af0c-473b-83f3-53edc6e91768-kube-api-access-n2g42\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.177738 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.177649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-proc\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.177738 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.177686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-lib-modules\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.177738 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.177720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-sys\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.278724 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-lib-modules\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.278907 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-sys\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.278907 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278831 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-sys\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.278907 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-lib-modules\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.279016 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-podres\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.279016 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.278948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2g42\" (UniqueName: \"kubernetes.io/projected/792bb765-af0c-473b-83f3-53edc6e91768-kube-api-access-n2g42\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.279083 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.279059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-proc\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.279126 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.279063 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-podres\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.279163 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.279130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792bb765-af0c-473b-83f3-53edc6e91768-proc\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.288324 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.288297 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2g42\" (UniqueName: \"kubernetes.io/projected/792bb765-af0c-473b-83f3-53edc6e91768-kube-api-access-n2g42\") pod \"perf-node-gather-daemonset-9ctjk\" (UID: \"792bb765-af0c-473b-83f3-53edc6e91768\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.409799 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.409706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:08.545958 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.545928 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk"] Jul 17 22:08:08.546705 ip-10-0-129-137 kubenswrapper[2573]: W0717 22:08:08.546683 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod792bb765_af0c_473b_83f3_53edc6e91768.slice/crio-22c5cb4dcd5a2e9c4d1b9ee1030273a92ed6024a2b3e2137b3a929321474872d WatchSource:0}: Error finding container 22c5cb4dcd5a2e9c4d1b9ee1030273a92ed6024a2b3e2137b3a929321474872d: Status 404 returned error can't find the container with id 22c5cb4dcd5a2e9c4d1b9ee1030273a92ed6024a2b3e2137b3a929321474872d Jul 17 22:08:08.941026 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:08.940944 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-775d944cb7-hpz2v_dd57b04c-f02f-4827-8a35-03abbc7198be/console/0.log" Jul 17 22:08:09.172245 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:09.172203 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" event={"ID":"792bb765-af0c-473b-83f3-53edc6e91768","Type":"ContainerStarted","Data":"ed0a5a3022a08b33217a601f929d0975de7d252a8748216cd17b29ba374f9bef"} Jul 17 22:08:09.172629 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:09.172261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" event={"ID":"792bb765-af0c-473b-83f3-53edc6e91768","Type":"ContainerStarted","Data":"22c5cb4dcd5a2e9c4d1b9ee1030273a92ed6024a2b3e2137b3a929321474872d"} Jul 17 22:08:09.172629 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:09.172283 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:09.188694 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:09.188649 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" podStartSLOduration=1.188635945 podStartE2EDuration="1.188635945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:09.186649252 +0000 UTC m=+3562.491648192" watchObservedRunningTime="2026-07-17 22:08:09.188635945 +0000 UTC m=+3562.493634876" Jul 17 22:08:10.620258 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:10.620229 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8nvrx_67f6c1d5-9932-4936-a7a3-18ccb9364ddd/dns/0.log" Jul 17 22:08:10.638740 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:10.638716 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8nvrx_67f6c1d5-9932-4936-a7a3-18ccb9364ddd/kube-rbac-proxy/0.log" Jul 17 22:08:10.719207 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:10.719186 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4ths_07f27250-1057-446a-99a5-e49eb4eb1354/dns-node-resolver/0.log" Jul 17 22:08:11.311959 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:11.311911 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cp4k9_08d39a33-6c3a-4dfb-a70e-9714ccc04802/node-ca/0.log" Jul 17 22:08:12.388045 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:12.388016 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-dw7vh_c57535d6-10b6-4d40-a748-f297653e7d91/kube-auth-proxy/0.log" Jul 17 22:08:12.434672 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:12.434642 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/0.log" Jul 17 22:08:12.470660 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:12.470634 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/1.log" Jul 17 22:08:12.498041 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:12.497988 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-86qlf_1fafc772-0bfc-433d-9462-e5922835c535/payload-pre-processing/0.log" Jul 17 22:08:12.525384 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:12.525351 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-72zq9_14411f37-d3fb-4c59-be8d-6662499ca5a9/payload-processing/0.log" Jul 17 22:08:13.251815 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:13.251785 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gg29x_d7fd4996-7d69-4e26-a89b-ca075b4b457e/serve-healthcheck-canary/0.log" Jul 17 22:08:13.823174 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:13.823098 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z6l26_99090a94-5102-43d5-911d-dd9aa6f8a288/kube-rbac-proxy/0.log" Jul 17 22:08:13.841172 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:13.841149 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z6l26_99090a94-5102-43d5-911d-dd9aa6f8a288/exporter/0.log" Jul 17 22:08:13.859383 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:13.859360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z6l26_99090a94-5102-43d5-911d-dd9aa6f8a288/extractor/0.log" Jul 17 22:08:15.187628 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:15.187601 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-9ctjk" Jul 17 22:08:16.357085 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:16.357058 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-557f54c68b-8gszz_4c4644f6-c066-4767-9727-07223eb066f5/manager/0.log" Jul 17 22:08:16.696583 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:16.696507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-vjsms_3fec7d27-e113-49aa-bf14-2767ee7782cf/manager/0.log" Jul 17 22:08:16.716241 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:16.716220 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-vjsms_3fec7d27-e113-49aa-bf14-2767ee7782cf/copy-manifests/0.log" Jul 17 22:08:18.214945 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.214862 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f487bdccb-btsvh_1f08f2cc-22ab-4acf-9181-0d89bf944c97/manager/0.log" Jul 17 22:08:18.310152 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.310122 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tbgsj_91caccb0-5692-4f1a-ae38-e6ee4b2c7ec9/openshift-lws-operator/0.log" Jul 17 22:08:18.845819 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.845795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-dw7vh_c57535d6-10b6-4d40-a748-f297653e7d91/kube-auth-proxy/0.log" Jul 17 22:08:18.878176 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.878147 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/0.log" Jul 17 22:08:18.908128 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.908101 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/1.log" Jul 17 22:08:18.926354 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.926334 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-86qlf_1fafc772-0bfc-433d-9462-e5922835c535/payload-pre-processing/0.log" Jul 17 22:08:18.944929 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:18.944907 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-72zq9_14411f37-d3fb-4c59-be8d-6662499ca5a9/payload-processing/0.log" Jul 17 22:08:20.273420 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:20.273396 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-dw7vh_c57535d6-10b6-4d40-a748-f297653e7d91/kube-auth-proxy/0.log" Jul 17 22:08:20.309445 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:20.309424 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/0.log" Jul 17 22:08:20.338129 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:20.338098 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8grmf_90ed55e8-e67e-407b-bf8b-e2fc121f47ca/istio-proxy/1.log" Jul 17 22:08:20.367612 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:20.367586 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-86qlf_1fafc772-0bfc-433d-9462-e5922835c535/payload-pre-processing/0.log" Jul 17 22:08:20.388603 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:20.388571 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-72zq9_14411f37-d3fb-4c59-be8d-6662499ca5a9/payload-processing/0.log" Jul 17 22:08:24.640980 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.640947 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/kube-multus-additional-cni-plugins/0.log" Jul 17 22:08:24.659486 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.659457 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/egress-router-binary-copy/0.log" Jul 17 22:08:24.676821 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.676803 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/cni-plugins/0.log" Jul 17 22:08:24.694138 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.694112 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/bond-cni-plugin/0.log" Jul 17 22:08:24.711511 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.711486 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/routeoverride-cni/0.log" Jul 17 22:08:24.729384 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.729362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/whereabouts-cni-bincopy/0.log" Jul 17 22:08:24.746805 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:24.746761 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cg54_cfda85cb-e637-42da-a583-f28076b66cd2/whereabouts-cni/0.log" Jul 17 22:08:25.183219 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:25.183193 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dpbg6_bbeba6e1-5ed1-44c5-a2d6-829f797c67f0/kube-multus/0.log" Jul 17 22:08:25.344303 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:25.344274 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hnhvt_20ef74bf-c023-43cb-90f1-b8db835926b4/network-metrics-daemon/0.log" Jul 17 22:08:25.362081 ip-10-0-129-137 kubenswrapper[2573]: I0717 22:08:25.362050 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hnhvt_20ef74bf-c023-43cb-90f1-b8db835926b4/kube-rbac-proxy/0.log"