Jul 07 18:48:12.883455 ip-10-0-132-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 18:48:12.883467 ip-10-0-132-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 18:48:12.883477 ip-10-0-132-43 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 18:48:12.883907 ip-10-0-132-43 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 18:48:24.168862 ip-10-0-132-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 18:48:24.168880 ip-10-0-132-43 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 18ad3d86b9fd4870824b144e1b6f1892 -- Jul 07 18:50:27.936953 ip-10-0-132-43 systemd[1]: Starting Kubernetes Kubelet... Jul 07 18:50:28.343390 ip-10-0-132-43 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:50:28.343390 ip-10-0-132-43 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 18:50:28.343390 ip-10-0-132-43 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:50:28.343390 ip-10-0-132-43 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 18:50:28.343390 ip-10-0-132-43 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:50:28.344908 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.344792 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 18:50:28.350273 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350255 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.350273 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350273 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350276 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350280 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350282 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350294 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350297 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350300 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350303 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350305 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350308 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350310 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350313 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350315 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350318 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350320 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350322 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350325 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350327 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350330 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350332 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.350340 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350334 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350337 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350341 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350343 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350346 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350349 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350352 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350355 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350358 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350360 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350363 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350365 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350368 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350370 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350373 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350378 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350380 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350383 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350385 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.350821 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350388 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350390 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350392 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350394 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350397 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350399 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350401 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350404 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350406 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350408 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350410 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350413 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350415 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350417 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350422 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350427 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350430 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350433 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350436 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.351270 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350440 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350444 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350447 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350450 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350452 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350455 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350458 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350460 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350462 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350465 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350468 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350471 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350474 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350476 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350478 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350481 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350483 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350485 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350488 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350491 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.351740 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350493 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350495 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350498 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350502 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350506 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350509 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350511 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350516 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350519 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350522 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350525 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350528 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350531 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350533 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350536 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350538 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350541 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350543 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350546 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.352203 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.350548 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351248 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351253 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351257 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351260 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351263 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351265 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351268 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351270 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351273 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351276 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351278 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351281 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351284 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351286 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351289 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351292 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351295 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351297 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351300 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.352641 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351302 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351305 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351308 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351310 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351313 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351316 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351318 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351321 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351323 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351325 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351327 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351330 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351332 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351335 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351337 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351340 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351342 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351346 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351348 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351351 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.353173 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351353 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351355 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351358 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351360 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351362 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351365 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351367 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351369 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351372 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351374 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351376 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351379 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351381 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351384 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351387 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351390 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351392 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351394 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351397 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351399 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.353637 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351402 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351404 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351406 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351409 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351411 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351414 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351417 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351419 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351421 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351424 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351426 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351429 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351440 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351444 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351448 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351450 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351453 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351456 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351458 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.354120 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351461 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351463 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351465 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351468 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351472 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351476 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351480 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351483 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351486 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351489 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351491 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351494 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351496 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351499 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351501 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351504 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351506 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351509 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351511 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.354561 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351515 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351518 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351897 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351903 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351907 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351909 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351912 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351915 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351918 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351920 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351924 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351927 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351929 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351932 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351934 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351937 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351940 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351942 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351945 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351947 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.355054 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351949 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351952 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351955 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351958 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351960 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351962 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351965 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351967 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351970 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351973 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351975 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351978 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351980 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351983 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351985 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351988 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351990 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351993 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351995 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.351997 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.355531 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352000 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352002 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352005 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352007 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352010 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352012 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352015 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352017 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352019 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352022 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352025 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352027 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352030 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352032 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352034 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352037 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352039 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352041 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352044 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352046 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.356016 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352048 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352051 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352053 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352057 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352060 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352063 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352066 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352070 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352073 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352076 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352079 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352081 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352086 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352089 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352091 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352093 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352098 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352100 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.356556 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352102 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352105 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352109 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352111 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352113 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352116 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352118 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352121 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352123 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352125 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352127 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352149 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352152 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352155 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352158 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352160 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352163 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352165 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352168 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.356996 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352170 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352173 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352175 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352177 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352245 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352252 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352259 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352265 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352269 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352273 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352277 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352282 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352285 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352288 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352291 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352294 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352297 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352300 2564 flags.go:64] FLAG: --cgroup-root="" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352303 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352306 2564 flags.go:64] FLAG: --client-ca-file="" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352308 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352311 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352315 2564 flags.go:64] FLAG: --cluster-domain="" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352317 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 18:50:28.357432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352320 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352323 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352326 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352330 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352333 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352335 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352338 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352341 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352344 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352347 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352350 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352352 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352356 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352359 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352362 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352367 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352369 2564 flags.go:64] FLAG: --enable-server="true" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352372 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352377 2564 flags.go:64] FLAG: --event-burst="100" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352380 2564 flags.go:64] FLAG: --event-qps="50" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352382 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352385 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352388 2564 flags.go:64] FLAG: --eviction-hard="" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352392 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 18:50:28.358052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352394 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352398 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352400 2564 flags.go:64] FLAG: --eviction-soft="" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352403 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352405 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352408 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352411 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352413 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352416 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352419 2564 flags.go:64] FLAG: --feature-gates="" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352425 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352428 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352430 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352433 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352436 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352439 2564 flags.go:64] FLAG: --help="false" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352441 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352444 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352447 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352450 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352453 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352456 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352459 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 18:50:28.358603 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352462 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352465 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352468 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352471 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352474 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352477 2564 flags.go:64] FLAG: --kube-reserved="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352480 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352482 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352485 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352488 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352491 2564 flags.go:64] FLAG: --lock-file="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352493 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352496 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352499 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352504 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352506 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352509 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352512 2564 flags.go:64] FLAG: --logging-format="text" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352514 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352517 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352520 2564 flags.go:64] FLAG: --manifest-url="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352523 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352526 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352530 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352534 2564 flags.go:64] FLAG: --max-pods="110" Jul 07 18:50:28.359178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352536 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352539 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352542 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352545 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352548 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352550 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352553 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352562 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352564 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352568 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352571 2564 flags.go:64] FLAG: --pod-cidr="" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352573 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352579 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352581 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352584 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352587 2564 flags.go:64] FLAG: --port="10250" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352590 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352593 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0124af6e28d18337d" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352595 2564 flags.go:64] FLAG: --qos-reserved="" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352598 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352601 2564 flags.go:64] FLAG: --register-node="true" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352603 2564 flags.go:64] FLAG: --register-with-taints="" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352607 2564 flags.go:64] FLAG: --registry-burst="10" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352609 2564 flags.go:64] FLAG: --registry-qps="5" Jul 07 18:50:28.359821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352612 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352614 2564 flags.go:64] FLAG: --reserved-memory="" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352617 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352620 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352623 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352626 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352628 2564 flags.go:64] FLAG: --runonce="false" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352631 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352633 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352636 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352639 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352642 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352659 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352662 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352665 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352670 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352672 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352675 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352678 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352681 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352684 2564 flags.go:64] FLAG: --system-cgroups="" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352687 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352692 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352694 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352697 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 18:50:28.360390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352701 2564 flags.go:64] FLAG: --tls-min-version="" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352704 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352706 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352709 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352712 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352714 2564 flags.go:64] FLAG: --v="2" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352718 2564 flags.go:64] FLAG: --version="false" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352722 2564 flags.go:64] FLAG: --vmodule="" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352725 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.352728 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352846 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352849 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352853 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352855 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352858 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352860 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352862 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352865 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352868 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352873 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352876 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.361160 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352879 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352883 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352885 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352888 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352890 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352893 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352896 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352898 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352901 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352903 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352905 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352908 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352910 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352912 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352915 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352917 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352920 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352922 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352924 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352926 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.361981 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352929 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352931 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352935 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352938 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352941 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352944 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352946 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352948 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352951 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352953 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352956 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352959 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352961 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352966 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352968 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352970 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352974 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352977 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352981 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.362639 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352983 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352986 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352988 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352991 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352993 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352996 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.352998 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353001 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353003 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353005 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353008 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353010 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353012 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353015 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353017 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353019 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353022 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353025 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353027 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353029 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.363122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353032 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353034 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353036 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353040 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353042 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353044 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353048 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353050 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353052 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353055 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353057 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353060 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353062 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353065 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353067 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353069 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353072 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353075 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353077 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353079 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.363601 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353081 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353084 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353086 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353089 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353091 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353093 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353096 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353099 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.353101 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.353890 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.360719 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.360735 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.360755 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.360764 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361067 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.364088 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361426 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361435 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361442 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361446 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361451 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361456 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361460 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361465 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361475 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361480 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361485 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361489 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361494 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361498 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361503 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361507 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361511 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361515 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361520 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361524 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.364459 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361533 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361537 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361541 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361547 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361553 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361557 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361561 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361566 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361573 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361578 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361582 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361586 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361595 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361599 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361603 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361608 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361612 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361616 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361620 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.364927 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361624 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361628 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361632 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361636 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361640 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361644 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361677 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361681 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361685 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361689 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361694 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361698 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361702 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361706 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361710 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361714 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361718 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361722 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361726 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361735 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.365427 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361739 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361744 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361749 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361753 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361756 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361760 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361765 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361769 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361774 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361778 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361782 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361790 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361795 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361799 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361803 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361807 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361811 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361815 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361820 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361823 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.365918 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361827 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361831 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361835 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361839 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361848 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361852 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361857 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361861 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361866 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361869 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361875 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361880 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361885 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361890 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361894 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361900 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361908 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361913 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.366392 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.361917 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.361925 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362330 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362345 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362349 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362352 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362354 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362357 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362360 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362363 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362365 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362368 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362372 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362377 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362380 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:50:28.367044 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362383 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362386 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362389 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362393 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362397 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362400 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362403 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362406 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362410 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362413 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362415 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362418 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362421 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362424 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362426 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362429 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362432 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362435 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362438 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:50:28.367408 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362442 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362445 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362448 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362452 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362454 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362457 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362460 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362462 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362465 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362467 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362469 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362472 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362474 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362476 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362479 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362482 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362484 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362487 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362489 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362492 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:50:28.367917 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362494 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362496 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362499 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362502 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362504 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362507 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362509 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362512 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362514 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362517 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362519 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362522 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362524 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362527 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362529 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362532 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362534 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362536 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362539 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362541 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:50:28.368528 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362543 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362546 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362548 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362551 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362553 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362556 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362559 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362561 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362564 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362566 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362571 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362573 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362576 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362578 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362581 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362583 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362586 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362589 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362591 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:50:28.369108 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362594 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362596 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362599 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362601 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362603 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362606 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362609 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:28.362611 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.362616 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.362773 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.365000 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.366459 2564 server.go:1019] "Starting client certificate rotation" Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.366575 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 18:50:28.369748 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.366619 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 18:50:28.370212 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.368835 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 18:50:28.372468 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.372447 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 18:50:28.386762 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.386737 2564 log.go:25] "Validated CRI v1 runtime API" Jul 07 18:50:28.390112 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.390093 2564 log.go:25] "Validated CRI v1 image API" Jul 07 18:50:28.390998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.390980 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 18:50:28.394854 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.394834 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bce7546e-5ae4-4cde-a66f-67e4fd3aeaad:/dev/nvme0n1p4 ec27ae42-d607-439b-8c52-955cfcf61aac:/dev/nvme0n1p3] Jul 07 18:50:28.394919 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.394855 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 18:50:28.399288 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.399271 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 18:50:28.400400 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.400285 2564 manager.go:216] Machine: {Timestamp:2026-07-07 18:50:28.398986503 +0000 UTC m=+0.341609281 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3203314 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8d7c5e3ade7780bf8fc26d92aa33 SystemUUID:ec2e8d7c-5e3a-de77-80bf-8fc26d92aa33 BootID:18ad3d86-b9fd-4870-824b-144e1b6f1892 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:8b:47:f3:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:8b:47:f3:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:1a:1c:3f:b8:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 18:50:28.400400 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.400399 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 18:50:28.400498 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.400488 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 18:50:28.404543 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.404515 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 18:50:28.404700 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.404546 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 18:50:28.405302 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.405292 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 18:50:28.405331 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.405305 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 18:50:28.405331 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.405312 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 18:50:28.408187 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.408173 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 18:50:28.409497 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.409486 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 18:50:28.412640 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412625 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 18:50:28.412797 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412787 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 18:50:28.412831 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412822 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 07 18:50:28.412866 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412846 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 18:50:28.412910 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412866 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 18:50:28.412910 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412882 2564 kubelet.go:398] "Adding apiserver pod source" Jul 07 18:50:28.412910 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.412895 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 18:50:28.413275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.413255 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 18:50:28.413332 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.413292 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 18:50:28.416979 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.416950 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 18:50:28.418987 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.418973 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 18:50:28.419066 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.419005 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 18:50:28.420178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420165 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 18:50:28.420222 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420182 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 18:50:28.420222 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420188 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 18:50:28.420222 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420194 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 18:50:28.420222 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420202 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 18:50:28.420222 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420207 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 18:50:28.420848 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420838 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 18:50:28.420883 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420850 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 18:50:28.420883 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420861 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 18:50:28.420883 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420869 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 18:50:28.420883 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420880 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 18:50:28.420980 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.420892 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 18:50:28.421906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.421896 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 18:50:28.421906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.421906 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 18:50:28.425225 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.425196 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 18:50:28.425302 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.425264 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 18:50:28.425363 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.425346 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 18:50:28.425617 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.425606 2564 server.go:1268] "Started kubelet" Jul 07 18:50:28.425708 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.425684 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 18:50:28.425765 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.425704 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 18:50:28.425829 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.425780 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 18:50:28.426050 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.426032 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76925" Jul 07 18:50:28.426686 ip-10-0-132-43 systemd[1]: Started Kubernetes Kubelet. Jul 07 18:50:28.427666 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.427608 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 18:50:28.427891 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.427879 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 07 18:50:28.430520 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.430501 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 18:50:28.433234 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.432384 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-43.ec2.internal.18c016889be550f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-43.ec2.internal,UID:ip-10-0-132-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-43.ec2.internal,},FirstTimestamp:2026-07-07 18:50:28.425576689 +0000 UTC m=+0.368199536,LastTimestamp:2026-07-07 18:50:28.425576689 +0000 UTC m=+0.368199536,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-43.ec2.internal,}" Jul 07 18:50:28.433357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.433344 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 18:50:28.433402 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.433349 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 18:50:28.433466 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.433448 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.434040 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434021 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 18:50:28.434128 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434035 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 18:50:28.434128 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434064 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 18:50:28.434215 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434197 2564 factory.go:55] Registering systemd factory Jul 07 18:50:28.434258 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434234 2564 factory.go:223] Registration of the systemd container factory successfully Jul 07 18:50:28.434300 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434198 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 07 18:50:28.434300 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434285 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 07 18:50:28.434521 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434504 2564 factory.go:153] Registering CRI-O factory Jul 07 18:50:28.434521 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434522 2564 factory.go:223] Registration of the crio container factory successfully Jul 07 18:50:28.434673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434561 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 18:50:28.434673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434586 2564 factory.go:103] Registering Raw factory Jul 07 18:50:28.434673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.434602 2564 manager.go:1196] Started watching for new ooms in manager Jul 07 18:50:28.435082 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.435071 2564 manager.go:349] Starting recovery of all containers Jul 07 18:50:28.436106 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.436076 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76925" Jul 07 18:50:28.438465 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.438435 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 18:50:28.438579 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.438504 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 18:50:28.444968 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.444946 2564 manager.go:354] Recovery completed Jul 07 18:50:28.450369 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.450357 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.452178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452161 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.452227 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452194 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.452227 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452208 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.452807 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452793 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 18:50:28.452807 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452801 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.452807 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452807 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 18:50:28.452929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.452825 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 18:50:28.453455 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453437 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.453545 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453469 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.453545 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453484 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.453943 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453926 2564 policy_none.go:49] "None policy: Start" Jul 07 18:50:28.453943 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453944 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 18:50:28.454059 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.453956 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 18:50:28.455162 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.455147 2564 policy_none.go:47] "Start" Jul 07 18:50:28.490263 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490243 2564 manager.go:337] "Starting Device Plugin manager" Jul 07 18:50:28.490389 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.490285 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 18:50:28.490389 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490298 2564 server.go:87] "Starting device plugin registration server" Jul 07 18:50:28.490601 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490587 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 18:50:28.490689 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490601 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 18:50:28.490733 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490712 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 18:50:28.490806 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490790 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 18:50:28.490806 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.490804 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 18:50:28.491413 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.491396 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 18:50:28.491480 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.491445 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.591153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.591111 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.592993 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.592973 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.593070 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.593013 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.593070 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.593027 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.593070 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.593055 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.604890 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.604851 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.604890 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.604872 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-43.ec2.internal\": node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.612753 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.612725 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 18:50:28.614089 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.614068 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 18:50:28.614179 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.614106 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 18:50:28.614179 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.614128 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 18:50:28.614179 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.614163 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 18:50:28.616458 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.616440 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:50:28.619607 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.619587 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.715036 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.714999 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal"] Jul 07 18:50:28.715190 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.715087 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.717497 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.717479 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.717573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.717513 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.717573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.717523 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.718862 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.718850 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.718990 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.718975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.719028 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719009 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.719631 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719616 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.719639 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719658 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719673 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719623 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719709 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.719718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.719718 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.720836 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.720824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.720894 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.720846 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:50:28.721497 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.721473 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:50:28.721497 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.721497 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:50:28.721636 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.721509 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:50:28.735600 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.735585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.735671 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.735609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84bdf0f4134502e8474ed50b3bda5122-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"84bdf0f4134502e8474ed50b3bda5122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.735671 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.735625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.749202 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.749182 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-43.ec2.internal\" not found" node="ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.753379 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.753363 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-43.ec2.internal\" not found" node="ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.819929 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.819904 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:28.836285 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836257 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.836374 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.836374 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.836374 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84bdf0f4134502e8474ed50b3bda5122-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"84bdf0f4134502e8474ed50b3bda5122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.836374 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836371 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84bdf0f4134502e8474ed50b3bda5122-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"84bdf0f4134502e8474ed50b3bda5122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.836491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:28.836394 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb67f3b88cc36fb2c1eb74425035048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"5fb67f3b88cc36fb2c1eb74425035048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:28.920884 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:28.920798 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.021589 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.021550 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.053141 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.053112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:29.057255 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.056881 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:29.122583 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.122548 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.223337 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.223268 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.323932 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.323901 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.365542 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.365503 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 18:50:29.365995 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.365693 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 18:50:29.425011 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.424974 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.437988 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.437945 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:45:28 +0000 UTC" deadline="2028-03-20 14:13:15.060586538 +0000 UTC" Jul 07 18:50:29.437988 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.437985 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14923h22m45.622604267s" Jul 07 18:50:29.525842 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.525761 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.626575 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.626547 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.651709 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:29.651679 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb67f3b88cc36fb2c1eb74425035048.slice/crio-f030c6e5102b3492990b24ce701553cd2adc43bb4f3cefbaf5268d83d0500e5e WatchSource:0}: Error finding container f030c6e5102b3492990b24ce701553cd2adc43bb4f3cefbaf5268d83d0500e5e: Status 404 returned error can't find the container with id f030c6e5102b3492990b24ce701553cd2adc43bb4f3cefbaf5268d83d0500e5e Jul 07 18:50:29.654318 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:29.654296 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bdf0f4134502e8474ed50b3bda5122.slice/crio-a3213fea86afae8211756286d8e6fab33dc8a7010b31b1f06b4c9e4bb33dcb5c WatchSource:0}: Error finding container a3213fea86afae8211756286d8e6fab33dc8a7010b31b1f06b4c9e4bb33dcb5c: Status 404 returned error can't find the container with id a3213fea86afae8211756286d8e6fab33dc8a7010b31b1f06b4c9e4bb33dcb5c Jul 07 18:50:29.658353 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.658338 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:50:29.726743 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:29.726707 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Jul 07 18:50:29.737176 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.737155 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:50:29.754262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.754240 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:50:29.834561 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.834492 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Jul 07 18:50:29.847532 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.847514 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 18:50:29.849280 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.849267 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Jul 07 18:50:29.859021 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.858998 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 18:50:29.956070 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:29.956045 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:50:30.414357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.414318 2564 apiserver.go:52] "Watching apiserver" Jul 07 18:50:30.423775 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.423746 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 18:50:30.425128 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.424609 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh","openshift-cluster-node-tuning-operator/tuned-f9zjp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal","openshift-multus/network-metrics-daemon-679vf","openshift-network-operator/iptables-alerter-cfdq4","kube-system/global-pull-secret-syncer-t68ch","openshift-dns/node-resolver-pbkl2","openshift-image-registry/node-ca-x26db","openshift-multus/multus-additional-cni-plugins-69rjl","openshift-multus/multus-cdbbp","openshift-network-diagnostics/network-check-target-2fk95","openshift-ovn-kubernetes/ovnkube-node-g75c6","kube-system/konnectivity-agent-m7zwd"] Jul 07 18:50:30.426516 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.426488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.428328 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.427943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.429352 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.429123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.430375 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430169 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 18:50:30.430375 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430310 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.430375 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.430575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430500 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 18:50:30.430575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vdkg8\"" Jul 07 18:50:30.430750 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430683 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.430750 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.430723 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 18:50:30.431447 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.431428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 18:50:30.431912 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.431893 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.432459 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.432441 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.432661 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.432614 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:30.432661 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.432642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6rdr\"" Jul 07 18:50:30.432865 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.432847 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.432957 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.432880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 18:50:30.432957 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.432938 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.433173 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433158 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 18:50:30.433173 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-24t8q\"" Jul 07 18:50:30.433341 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.433341 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433241 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p7l6n\"" Jul 07 18:50:30.434585 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433747 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 18:50:30.434585 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.433977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.436007 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.435243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:30.436007 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.435309 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:30.436173 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.436156 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.436244 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.436222 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 18:50:30.436485 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.436470 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 18:50:30.437275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.437257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.437547 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.437508 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 18:50:30.437838 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.437822 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 18:50:30.438136 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.438121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sprrl\"" Jul 07 18:50:30.438347 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.438331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.438457 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.438422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.440381 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.440359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 18:50:30.440514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.440498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.440787 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.440769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lcbr5\"" Jul 07 18:50:30.441436 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.441393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 18:50:30.441436 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.441421 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.441523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.441463 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.441718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.441702 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d86d4\"" Jul 07 18:50:30.442484 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.442156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.442484 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.442332 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:30.442619 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.442603 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.442906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.442891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xvwfr\"" Jul 07 18:50:30.443081 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.443066 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.443924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.443499 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.446104 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysconfig\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446185 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-systemd-units\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446185 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-ovn\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446185 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902cac6c-1abe-40c6-83f1-33f3f212dae0-konnectivity-ca\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-registration-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-tmp\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2nv\" (UniqueName: \"kubernetes.io/projected/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-kube-api-access-zv2nv\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902cac6c-1abe-40c6-83f1-33f3f212dae0-agent-certs\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-var-lib-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-host\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gdf\" (UniqueName: \"kubernetes.io/projected/01107935-02af-4824-a87a-2d9052e0e239-kube-api-access-45gdf\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-etc-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-node-log\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-conf\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.446612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-config\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-sys-fs\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-socket-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-kubernetes\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-sys\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-netd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-lib-modules\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-env-overrides\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-var-lib-kubelet\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446948 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fdngd\"" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-log-socket\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.446992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-bin\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-script-lib\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447030 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-systemd\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-tuned\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-os-release\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwhd\" (UniqueName: \"kubernetes.io/projected/0381d901-b528-4f36-98b7-b82be8f1b698-kube-api-access-qrwhd\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447200 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-device-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-modprobe-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kbv\" (UniqueName: \"kubernetes.io/projected/e59d0e33-9a13-4d3a-9f93-eb637735d72f-kube-api-access-c5kbv\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vrq\" (UniqueName: \"kubernetes.io/projected/73529dd4-3aa9-483a-8a15-bc908e780f23-kube-api-access-s8vrq\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-cnibin\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-kubelet\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-slash\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-run\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.447759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-system-cni-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-binary-copy\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-netns\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-systemd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 18:50:30.448407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.447576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovn-node-metrics-cert\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.451563 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.451539 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 18:50:30.494435 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.494407 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jqhkn" Jul 07 18:50:30.505785 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.505761 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jqhkn" Jul 07 18:50:30.535038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.535010 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 18:50:30.548803 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-log-socket\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.548929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-bin\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.548929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-script-lib\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.548929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-systemd\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.548929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-tuned\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.548929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-multus-certs\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-os-release\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.548980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwhd\" (UniqueName: \"kubernetes.io/projected/0381d901-b528-4f36-98b7-b82be8f1b698-kube-api-access-qrwhd\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-device-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-modprobe-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-multus\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-dbus\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kbv\" (UniqueName: \"kubernetes.io/projected/e59d0e33-9a13-4d3a-9f93-eb637735d72f-kube-api-access-c5kbv\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vrq\" (UniqueName: \"kubernetes.io/projected/73529dd4-3aa9-483a-8a15-bc908e780f23-kube-api-access-s8vrq\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-daemon-config\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96182dfd-28d7-4102-89e8-73269420dc2d-serviceca\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-cnibin\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-kubelet\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-slash\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-run\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-conf-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-kubelet-config\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-system-cni-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-binary-copy\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-netns\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-systemd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovn-node-metrics-cert\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysconfig\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.549801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-socket-dir-parent\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsjj\" (UniqueName: \"kubernetes.io/projected/1b298066-20c5-4300-8f0b-d3bab2a0a24c-kube-api-access-vjsjj\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-systemd-units\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-ovn\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902cac6c-1abe-40c6-83f1-33f3f212dae0-konnectivity-ca\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70b3cc77-6a87-493d-90f0-880940f2a64d-host-slash\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-bin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-hostroot\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96182dfd-28d7-4102-89e8-73269420dc2d-host\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-registration-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.549986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-tmp\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2nv\" (UniqueName: \"kubernetes.io/projected/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-kube-api-access-zv2nv\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902cac6c-1abe-40c6-83f1-33f3f212dae0-agent-certs\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-var-lib-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-host\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.550558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cnibin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-os-release\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gdf\" (UniqueName: \"kubernetes.io/projected/01107935-02af-4824-a87a-2d9052e0e239-kube-api-access-45gdf\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-etc-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-node-log\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-conf\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-netns\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrb7\" (UniqueName: \"kubernetes.io/projected/96182dfd-28d7-4102-89e8-73269420dc2d-kube-api-access-bfrb7\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-k8s-cni-cncf-io\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-etc-kubernetes\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwnk\" (UniqueName: \"kubernetes.io/projected/3c185cd3-fd4e-499d-be64-bbdc0a43357a-kube-api-access-zbwnk\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-config\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-sys-fs\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cni-binary-copy\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298066-20c5-4300-8f0b-d3bab2a0a24c-tmp-dir\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.551333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-socket-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-kubernetes\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-sys\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-kubelet\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-netd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-lib-modules\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/70b3cc77-6a87-493d-90f0-880940f2a64d-kube-api-access-8fc5l\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-system-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-env-overrides\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-var-lib-kubelet\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70b3cc77-6a87-493d-90f0-880940f2a64d-iptables-alerter-script\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.550971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b298066-20c5-4300-8f0b-d3bab2a0a24c-hosts-file\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.551072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-log-socket\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.551115 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-bin\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.551753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-script-lib\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.552863 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.551846 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-systemd\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.552863 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.552109 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 18:50:30.552863 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.552512 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.553002 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.552870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-os-release\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.553117 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553098 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.553228 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553206 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.553299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553285 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-device-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.553378 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553363 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-modprobe-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.553787 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553748 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-registration-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.553881 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553821 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.553881 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.553860 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-d\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.554272 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554244 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-cnibin\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.554357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554298 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-kubelet\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.554357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-slash\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.554455 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-run\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.554751 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554718 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.554820 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.554785 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01107935-02af-4824-a87a-2d9052e0e239-system-cni-dir\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.555255 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555225 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-binary-copy\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.555334 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555292 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-run-netns\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.555372 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555333 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-systemd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.555786 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555765 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-run-ovn\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.555868 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysconfig\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.555974 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.555903 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-systemd-units\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.556855 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.556835 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-tmp\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.556981 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.556962 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:30.557053 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.557039 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:31.057020095 +0000 UTC m=+2.999642875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovnkube-config\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557631 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-sys-fs\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557636 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e59d0e33-9a13-4d3a-9f93-eb637735d72f-ovn-node-metrics-cert\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557759 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-tuned\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557848 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73529dd4-3aa9-483a-8a15-bc908e780f23-socket-dir\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-host-cni-netd\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-kubernetes\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.557964 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-lib-modules\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-var-lib-kubelet\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558070 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902cac6c-1abe-40c6-83f1-33f3f212dae0-konnectivity-ca\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558089 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-etc-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558139 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-host\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558147 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-var-lib-openvswitch\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558194 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e59d0e33-9a13-4d3a-9f93-eb637735d72f-node-log\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558243 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e59d0e33-9a13-4d3a-9f93-eb637735d72f-env-overrides\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.558368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558295 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-etc-sysctl-conf\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.559005 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558344 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-sys\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.559005 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.558535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01107935-02af-4824-a87a-2d9052e0e239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.559121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.559102 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902cac6c-1abe-40c6-83f1-33f3f212dae0-agent-certs\") pod \"konnectivity-agent-m7zwd\" (UID: \"902cac6c-1abe-40c6-83f1-33f3f212dae0\") " pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.559691 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.559597 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:30.559691 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.559622 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:30.559691 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.559636 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:30.559860 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.559703 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:31.059686973 +0000 UTC m=+3.002309758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:30.561995 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.561972 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwhd\" (UniqueName: \"kubernetes.io/projected/0381d901-b528-4f36-98b7-b82be8f1b698-kube-api-access-qrwhd\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:30.562368 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.562348 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vrq\" (UniqueName: \"kubernetes.io/projected/73529dd4-3aa9-483a-8a15-bc908e780f23-kube-api-access-s8vrq\") pod \"aws-ebs-csi-driver-node-zhdfh\" (UID: \"73529dd4-3aa9-483a-8a15-bc908e780f23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.562769 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.562751 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kbv\" (UniqueName: \"kubernetes.io/projected/e59d0e33-9a13-4d3a-9f93-eb637735d72f-kube-api-access-c5kbv\") pod \"ovnkube-node-g75c6\" (UID: \"e59d0e33-9a13-4d3a-9f93-eb637735d72f\") " pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.569156 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.569135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2nv\" (UniqueName: \"kubernetes.io/projected/dd8746b2-4e0d-43a4-9b97-d680adc5c6f0-kube-api-access-zv2nv\") pod \"tuned-f9zjp\" (UID: \"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0\") " pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.569894 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.569875 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gdf\" (UniqueName: \"kubernetes.io/projected/01107935-02af-4824-a87a-2d9052e0e239-kube-api-access-45gdf\") pod \"multus-additional-cni-plugins-69rjl\" (UID: \"01107935-02af-4824-a87a-2d9052e0e239\") " pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.619297 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.619238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" event={"ID":"84bdf0f4134502e8474ed50b3bda5122","Type":"ContainerStarted","Data":"a3213fea86afae8211756286d8e6fab33dc8a7010b31b1f06b4c9e4bb33dcb5c"} Jul 07 18:50:30.620910 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.620881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"5fb67f3b88cc36fb2c1eb74425035048","Type":"ContainerStarted","Data":"f030c6e5102b3492990b24ce701553cd2adc43bb4f3cefbaf5268d83d0500e5e"} Jul 07 18:50:30.651243 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-conf-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651243 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-kubelet-config\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-socket-dir-parent\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsjj\" (UniqueName: \"kubernetes.io/projected/1b298066-20c5-4300-8f0b-d3bab2a0a24c-kube-api-access-vjsjj\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70b3cc77-6a87-493d-90f0-880940f2a64d-host-slash\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-conf-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-bin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-hostroot\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96182dfd-28d7-4102-89e8-73269420dc2d-host\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-socket-dir-parent\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cnibin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-os-release\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651446 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-kubelet-config\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-netns\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrb7\" (UniqueName: \"kubernetes.io/projected/96182dfd-28d7-4102-89e8-73269420dc2d-kube-api-access-bfrb7\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651487 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-hostroot\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-k8s-cni-cncf-io\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-etc-kubernetes\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwnk\" (UniqueName: \"kubernetes.io/projected/3c185cd3-fd4e-499d-be64-bbdc0a43357a-kube-api-access-zbwnk\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cni-binary-copy\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298066-20c5-4300-8f0b-d3bab2a0a24c-tmp-dir\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-kubelet\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/70b3cc77-6a87-493d-90f0-880940f2a64d-kube-api-access-8fc5l\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-system-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70b3cc77-6a87-493d-90f0-880940f2a64d-iptables-alerter-script\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b298066-20c5-4300-8f0b-d3bab2a0a24c-hosts-file\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651783 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70b3cc77-6a87-493d-90f0-880940f2a64d-host-slash\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.651805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-multus-certs\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-multus-certs\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-multus\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651890 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96182dfd-28d7-4102-89e8-73269420dc2d-host\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-dbus\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651925 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cnibin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-daemon-config\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96182dfd-28d7-4102-89e8-73269420dc2d-serviceca\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651989 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-netns\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.651962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-os-release\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652262 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652023 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-run-k8s-cni-cncf-io\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-bin\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652573 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.652361 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:30.652573 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:30.652400 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:31.152387161 +0000 UTC m=+3.095009926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:30.652573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652424 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-cni-binary-copy\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-cni-multus\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652589 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298066-20c5-4300-8f0b-d3bab2a0a24c-tmp-dir\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.652736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652620 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-host-var-lib-kubelet\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652711 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41b38953-6cb3-4893-ac18-51c6f63c818d-dbus\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:30.652810 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652794 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-system-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652913 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652888 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-etc-kubernetes\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652986 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652928 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-cni-dir\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.652986 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.652939 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b298066-20c5-4300-8f0b-d3bab2a0a24c-hosts-file\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.653169 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.653153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c185cd3-fd4e-499d-be64-bbdc0a43357a-multus-daemon-config\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.653300 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.653278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70b3cc77-6a87-493d-90f0-880940f2a64d-iptables-alerter-script\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.653372 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.653321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96182dfd-28d7-4102-89e8-73269420dc2d-serviceca\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.672678 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.672468 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrb7\" (UniqueName: \"kubernetes.io/projected/96182dfd-28d7-4102-89e8-73269420dc2d-kube-api-access-bfrb7\") pod \"node-ca-x26db\" (UID: \"96182dfd-28d7-4102-89e8-73269420dc2d\") " pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.676086 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.676054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsjj\" (UniqueName: \"kubernetes.io/projected/1b298066-20c5-4300-8f0b-d3bab2a0a24c-kube-api-access-vjsjj\") pod \"node-resolver-pbkl2\" (UID: \"1b298066-20c5-4300-8f0b-d3bab2a0a24c\") " pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.676210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.676099 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwnk\" (UniqueName: \"kubernetes.io/projected/3c185cd3-fd4e-499d-be64-bbdc0a43357a-kube-api-access-zbwnk\") pod \"multus-cdbbp\" (UID: \"3c185cd3-fd4e-499d-be64-bbdc0a43357a\") " pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.676639 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.676615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/70b3cc77-6a87-493d-90f0-880940f2a64d-kube-api-access-8fc5l\") pod \"iptables-alerter-cfdq4\" (UID: \"70b3cc77-6a87-493d-90f0-880940f2a64d\") " pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:30.739803 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.739765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-69rjl" Jul 07 18:50:30.749210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.749185 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" Jul 07 18:50:30.769205 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.769175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" Jul 07 18:50:30.775104 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.775082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:30.782009 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.781985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:30.790071 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.790048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cdbbp" Jul 07 18:50:30.797767 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.797748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x26db" Jul 07 18:50:30.802523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.802506 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:50:30.805679 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.805660 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pbkl2" Jul 07 18:50:30.813330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:30.813312 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfdq4" Jul 07 18:50:31.155680 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:31.155637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:31.155823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:31.155704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:31.155823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:31.155736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:31.155823 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155803 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155839 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155861 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:32.155845996 +0000 UTC m=+4.098468761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155868 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155886 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:32.155870726 +0000 UTC m=+4.098493504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155890 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:31.155933 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155903 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:31.156130 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:31.155952 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:32.155938057 +0000 UTC m=+4.098560833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:31.507940 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:31.507841 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:45:30 +0000 UTC" deadline="2028-02-19 09:55:46.629627283 +0000 UTC" Jul 07 18:50:31.507940 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:31.507876 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14199h5m15.121754914s" Jul 07 18:50:31.694471 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:31.694442 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96182dfd_28d7_4102_89e8_73269420dc2d.slice/crio-26c587c6db50cd2d082e3664ee3a6568381cd91a5c29fb30c8101ca191d7f94c WatchSource:0}: Error finding container 26c587c6db50cd2d082e3664ee3a6568381cd91a5c29fb30c8101ca191d7f94c: Status 404 returned error can't find the container with id 26c587c6db50cd2d082e3664ee3a6568381cd91a5c29fb30c8101ca191d7f94c Jul 07 18:50:31.695113 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:31.695093 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c185cd3_fd4e_499d_be64_bbdc0a43357a.slice/crio-95eceabf4043515e659fd7b086a55657aaad3de9a19ae8b634793067adc8269b WatchSource:0}: Error finding container 95eceabf4043515e659fd7b086a55657aaad3de9a19ae8b634793067adc8269b: Status 404 returned error can't find the container with id 95eceabf4043515e659fd7b086a55657aaad3de9a19ae8b634793067adc8269b Jul 07 18:50:31.696251 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:31.696138 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59d0e33_9a13_4d3a_9f93_eb637735d72f.slice/crio-b523f4f95d3d334527842c536650b9affc4a7878a5b017ab3010d720e386fbc5 WatchSource:0}: Error finding container b523f4f95d3d334527842c536650b9affc4a7878a5b017ab3010d720e386fbc5: Status 404 returned error can't find the container with id b523f4f95d3d334527842c536650b9affc4a7878a5b017ab3010d720e386fbc5 Jul 07 18:50:31.696862 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:31.696845 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8746b2_4e0d_43a4_9b97_d680adc5c6f0.slice/crio-2df5f67abf8f75698a74bb08642c0c029099deafdb4c9f0950697053e65defef WatchSource:0}: Error finding container 2df5f67abf8f75698a74bb08642c0c029099deafdb4c9f0950697053e65defef: Status 404 returned error can't find the container with id 2df5f67abf8f75698a74bb08642c0c029099deafdb4c9f0950697053e65defef Jul 07 18:50:31.707082 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:50:31.707058 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b298066_20c5_4300_8f0b_d3bab2a0a24c.slice/crio-bd7c5a97295e1e96207a473ba442dedf515d2333cd4f06ebd77f3e343f9ef3ca WatchSource:0}: Error finding container bd7c5a97295e1e96207a473ba442dedf515d2333cd4f06ebd77f3e343f9ef3ca: Status 404 returned error can't find the container with id bd7c5a97295e1e96207a473ba442dedf515d2333cd4f06ebd77f3e343f9ef3ca Jul 07 18:50:32.162104 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.161836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:32.162104 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.162098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162011 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.162131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162184 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:34.162169746 +0000 UTC m=+6.104792539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162236 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162244 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162265 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162278 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162292 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:34.162276437 +0000 UTC m=+6.104899214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:32.162427 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.162322 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:34.162308422 +0000 UTC m=+6.104931188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:32.508953 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.508750 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:45:30 +0000 UTC" deadline="2028-03-16 22:01:34.92113239 +0000 UTC" Jul 07 18:50:32.508953 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.508823 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14835h11m2.412314328s" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.617313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.617450 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.617943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.618032 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.618090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:32.618235 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:32.618150 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:32.633021 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.632990 2564 generic.go:356] "Generic (PLEG): container finished" podID="5fb67f3b88cc36fb2c1eb74425035048" containerID="c3d8b94e88410517f111f65f13ca9d4011ffcb4047d0b3298c4e8758e614ebf2" exitCode=0 Jul 07 18:50:32.633186 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.633092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"5fb67f3b88cc36fb2c1eb74425035048","Type":"ContainerDied","Data":"c3d8b94e88410517f111f65f13ca9d4011ffcb4047d0b3298c4e8758e614ebf2"} Jul 07 18:50:32.640278 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.640224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pbkl2" event={"ID":"1b298066-20c5-4300-8f0b-d3bab2a0a24c","Type":"ContainerStarted","Data":"bd7c5a97295e1e96207a473ba442dedf515d2333cd4f06ebd77f3e343f9ef3ca"} Jul 07 18:50:32.645113 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.645049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfdq4" event={"ID":"70b3cc77-6a87-493d-90f0-880940f2a64d","Type":"ContainerStarted","Data":"695bdf2f59dc25fc1a8fdc9eee87c5b46e6c1438baf7cf83312a5c5a559a4a9c"} Jul 07 18:50:32.647192 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.647144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7zwd" event={"ID":"902cac6c-1abe-40c6-83f1-33f3f212dae0","Type":"ContainerStarted","Data":"88b4ca82242e01f58b4730f0b90b0ced3069bce975e2df0001853806a0cf5ec0"} Jul 07 18:50:32.651380 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.651292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" event={"ID":"73529dd4-3aa9-483a-8a15-bc908e780f23","Type":"ContainerStarted","Data":"4196807ae2208c09fc70861ce632002ebec3b8664d595e949235fa97ff1d183d"} Jul 07 18:50:32.661809 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.661781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerStarted","Data":"6cceea35382b6e3eac62784eec4798cc9309a9eb780614153395824f61ec5030"} Jul 07 18:50:32.678901 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.678868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"b523f4f95d3d334527842c536650b9affc4a7878a5b017ab3010d720e386fbc5"} Jul 07 18:50:32.694353 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.694318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" event={"ID":"84bdf0f4134502e8474ed50b3bda5122","Type":"ContainerStarted","Data":"2bb588f00a2421cebec3f6248b86f35b5189014b9119e3b95462e22572e6d2ca"} Jul 07 18:50:32.697684 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.697635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" event={"ID":"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0","Type":"ContainerStarted","Data":"2df5f67abf8f75698a74bb08642c0c029099deafdb4c9f0950697053e65defef"} Jul 07 18:50:32.702014 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.701919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cdbbp" event={"ID":"3c185cd3-fd4e-499d-be64-bbdc0a43357a","Type":"ContainerStarted","Data":"95eceabf4043515e659fd7b086a55657aaad3de9a19ae8b634793067adc8269b"} Jul 07 18:50:32.705759 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.705683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x26db" event={"ID":"96182dfd-28d7-4102-89e8-73269420dc2d","Type":"ContainerStarted","Data":"26c587c6db50cd2d082e3664ee3a6568381cd91a5c29fb30c8101ca191d7f94c"} Jul 07 18:50:32.711376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:32.711251 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" podStartSLOduration=3.711232899 podStartE2EDuration="3.711232899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:32.710341459 +0000 UTC m=+4.652964247" watchObservedRunningTime="2026-07-07 18:50:32.711232899 +0000 UTC m=+4.653855689" Jul 07 18:50:33.725377 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:33.725167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"5fb67f3b88cc36fb2c1eb74425035048","Type":"ContainerStarted","Data":"cdf15f64b00aa7f7d0a59f37f2d0352f1d043a7b595dafe72c5f40b77da05f44"} Jul 07 18:50:33.742631 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:33.741995 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" podStartSLOduration=4.741965997 podStartE2EDuration="4.741965997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:33.74136835 +0000 UTC m=+5.683991138" watchObservedRunningTime="2026-07-07 18:50:33.741965997 +0000 UTC m=+5.684588783" Jul 07 18:50:34.179988 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.179833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:34.179988 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.179983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:34.180200 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.180019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:34.180200 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180154 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:34.180305 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180213 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:38.180195494 +0000 UTC m=+10.122818275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:34.180603 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180585 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:34.180724 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180635 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:38.180621838 +0000 UTC m=+10.123244616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:34.180783 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180732 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:34.180783 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180750 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:34.180783 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180762 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:34.180930 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.180799 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:38.180787447 +0000 UTC m=+10.123410218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:34.614851 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.614775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:34.614851 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.614807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:34.615059 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.614903 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:34.615059 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:34.614940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:34.615059 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.615035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:34.615220 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:34.615124 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:36.615862 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:36.615785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:36.616276 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:36.615926 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:36.616276 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:36.615785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:36.616276 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:36.616016 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:36.616276 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:36.616037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:36.616276 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:36.616099 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:38.212901 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.212865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.212926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.212955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213027 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213041 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213092 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:46.213073449 +0000 UTC m=+18.155696228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213096 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213120 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213134 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213107 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:46.21310094 +0000 UTC m=+18.155723705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:38.213350 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.213195 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:46.21317917 +0000 UTC m=+18.155801940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:38.615549 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.615459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:38.615719 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.615578 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:38.615719 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.615592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:38.615719 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:38.615713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:38.615875 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.615737 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:38.615875 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:38.615813 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:40.615178 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:40.615149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:40.615555 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:40.615153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:40.615555 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:40.615263 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:40.615555 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:40.615286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:40.615555 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:40.615375 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:40.615555 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:40.615447 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.614820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:42.614949 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.615325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:42.615421 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.615489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:42.615610 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:42.615571 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:42.746403 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.746320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x26db" event={"ID":"96182dfd-28d7-4102-89e8-73269420dc2d","Type":"ContainerStarted","Data":"72f18e4342f3614c94e564d7ddd99d4a7c2a52b2d7992b38e60bbeedc472f7b5"} Jul 07 18:50:42.747842 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.747813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pbkl2" event={"ID":"1b298066-20c5-4300-8f0b-d3bab2a0a24c","Type":"ContainerStarted","Data":"a13898c23473d109b37d6c10505fc414e85d9e66945b77f777639eb8a9f57ec9"} Jul 07 18:50:42.749367 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.749341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7zwd" event={"ID":"902cac6c-1abe-40c6-83f1-33f3f212dae0","Type":"ContainerStarted","Data":"5f8103613782c6cc519065dd7d100bd61532e7177be6c2419f8a2ec695889647"} Jul 07 18:50:42.750764 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.750742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" event={"ID":"73529dd4-3aa9-483a-8a15-bc908e780f23","Type":"ContainerStarted","Data":"3f6aa145d395b46995a4f47297150633de67cf01294e1b6f98917ec5825d9d7b"} Jul 07 18:50:42.752291 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.752267 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="de089e0c67cc60d6a9a4f885812f72ff701ea8585713a3e2fd4fd351d0871258" exitCode=0 Jul 07 18:50:42.752378 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.752303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"de089e0c67cc60d6a9a4f885812f72ff701ea8585713a3e2fd4fd351d0871258"} Jul 07 18:50:42.753842 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.753822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" event={"ID":"dd8746b2-4e0d-43a4-9b97-d680adc5c6f0","Type":"ContainerStarted","Data":"e19240e0a654d85ba3f53bac66138194adee02f71833e8f7e1e19f2cec619f67"} Jul 07 18:50:42.762022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.761976 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x26db" podStartSLOduration=4.775591697 podStartE2EDuration="14.761960403s" totalImagesPullingTime="9.986368706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:42.761608775 +0000 UTC m=+14.704231561" watchObservedRunningTime="2026-07-07 18:50:42.761960403 +0000 UTC m=+14.704583189" Jul 07 18:50:42.779541 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.779500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m7zwd" podStartSLOduration=4.822036872 podStartE2EDuration="14.779488103s" totalImagesPullingTime="9.957451231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:42.779256726 +0000 UTC m=+14.721879519" watchObservedRunningTime="2026-07-07 18:50:42.779488103 +0000 UTC m=+14.722110889" Jul 07 18:50:42.795776 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.795734 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pbkl2" podStartSLOduration=3.816313692 podStartE2EDuration="13.795721852s" totalImagesPullingTime="9.97940816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:42.795531344 +0000 UTC m=+14.738154131" watchObservedRunningTime="2026-07-07 18:50:42.795721852 +0000 UTC m=+14.738344639" Jul 07 18:50:42.838336 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.838112 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f9zjp" podStartSLOduration=4.719357067 podStartE2EDuration="14.838091892s" totalImagesPullingTime="10.118734825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:42.837888203 +0000 UTC m=+14.780510991" watchObservedRunningTime="2026-07-07 18:50:42.838091892 +0000 UTC m=+14.780714679" Jul 07 18:50:42.926284 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.926242 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:42.926797 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:42.926769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:43.225430 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:43.225407 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x26db_96182dfd-28d7-4102-89e8-73269420dc2d/node-ca/0.log" Jul 07 18:50:43.757240 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:43.757197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfdq4" event={"ID":"70b3cc77-6a87-493d-90f0-880940f2a64d","Type":"ContainerStarted","Data":"db483e061ba467a8bb979b4c966da08c54d7335469360649fa1abfd1f1d62c3c"} Jul 07 18:50:43.758234 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:43.758213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:43.758720 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:43.758701 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m7zwd" Jul 07 18:50:43.792540 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:43.791608 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cfdq4" podStartSLOduration=4.836106957 podStartE2EDuration="14.791587782s" totalImagesPullingTime="9.955480825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:43.775240579 +0000 UTC m=+15.717863368" watchObservedRunningTime="2026-07-07 18:50:43.791587782 +0000 UTC m=+15.734210570" Jul 07 18:50:44.615209 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:44.615180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:44.615382 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:44.615179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:44.615382 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:44.615305 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:44.615382 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:44.615190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:44.615561 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:44.615377 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:44.615561 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:44.615473 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:46.277203 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.277162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.277227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.277254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277326 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277326 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277392 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:02.277376556 +0000 UTC m=+34.219999341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277406 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:51:02.277400058 +0000 UTC m=+34.220022823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277409 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277439 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277453 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:46.277924 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.277510 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:02.277492012 +0000 UTC m=+34.220114777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:50:46.614718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.614624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:46.614889 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.614624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:46.614889 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.614763 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:46.614889 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.614865 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:46.615027 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:46.614930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:46.615027 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:46.614996 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:48.616139 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:48.616104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:48.616819 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:48.616281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:48.616819 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:48.616352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:48.616819 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:48.616466 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:48.616967 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:48.616924 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:48.617075 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:48.617055 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:50.614844 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:50.614812 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:50.614844 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:50.614835 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:50.615456 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:50.614813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:50.615456 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:50.614927 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:50.615456 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:50.615029 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:50.615456 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:50.615128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:52.617793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:52.617761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:52.617793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:52.617779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:52.617793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:52.617761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:52.618249 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:52.617889 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:52.618249 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:52.617917 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:52.618249 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:52.618034 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:53.337075 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.336877 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 18:50:53.509610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.509518 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T18:50:53.337067805Z","UUID":"2495d999-0722-4b9e-be17-a5ce92a3d018","Handler":null,"Name":"","Endpoint":""} Jul 07 18:50:53.511201 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.511183 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 18:50:53.511302 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.511219 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 18:50:53.780219 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.779985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" event={"ID":"73529dd4-3aa9-483a-8a15-bc908e780f23","Type":"ContainerStarted","Data":"27274a43e95367d96f63152ccbae733bddf9df0f7c4f4f08c8057ce4236b9755"} Jul 07 18:50:53.781876 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.781847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerStarted","Data":"e09c91a8e975f3220211710912eff7d642952fcfe7b03caea1baa3cffb815e1e"} Jul 07 18:50:53.783998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.783976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"07897d70616d9a0feaf91ce7cea7a32417db99ddb2dd8ed526480e43d1825d43"} Jul 07 18:50:53.784092 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.784053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"60b6edc73bf3810d89f5e7d6b74a1a3d07face755ed0d0d1634e461bdf1aff43"} Jul 07 18:50:53.784137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.784098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"a0d4fd86ea0a8bed54061e9b34d75e11a9ffee923507004626238a5116d33931"} Jul 07 18:50:53.785286 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.785265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cdbbp" event={"ID":"3c185cd3-fd4e-499d-be64-bbdc0a43357a","Type":"ContainerStarted","Data":"17d5d847afcf9458e2260290de628e598b3acc0f8b845d439ef73112e0bb355d"} Jul 07 18:50:53.820258 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:53.820216 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cdbbp" podStartSLOduration=4.330617751 podStartE2EDuration="25.820200207s" totalImagesPullingTime="21.489582456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:53.82008631 +0000 UTC m=+25.762709099" watchObservedRunningTime="2026-07-07 18:50:53.820200207 +0000 UTC m=+25.762822994" Jul 07 18:50:54.618206 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.618175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:54.618454 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.618276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:54.618454 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:54.618279 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:54.618454 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.618283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:54.618454 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:54.618368 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:54.618454 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:54.618407 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:54.789011 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.788981 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="e09c91a8e975f3220211710912eff7d642952fcfe7b03caea1baa3cffb815e1e" exitCode=0 Jul 07 18:50:54.789427 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.789052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"e09c91a8e975f3220211710912eff7d642952fcfe7b03caea1baa3cffb815e1e"} Jul 07 18:50:54.791994 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.791973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"a3ad61f3f34d6b5e885ac46f2ea2dd6312ebc283ac06ffa52df952608c5dfae0"} Jul 07 18:50:54.792077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.792046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"acf65830b6ca975b8d36e0c1faf544df9a1a3529bcae40b8a09c4027b10efdc2"} Jul 07 18:50:54.792121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:54.792078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"6afc87e932c79e455d07d2394b1e3f85ef00812defcb816be6e52e5efd029972"} Jul 07 18:50:55.795489 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:55.795457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" event={"ID":"73529dd4-3aa9-483a-8a15-bc908e780f23","Type":"ContainerStarted","Data":"c2866ebd41b62abd55ac25e93f1d2a2071b69017f7d7004f0bcd14fa92e53a39"} Jul 07 18:50:55.816480 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:55.816432 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhdfh" podStartSLOduration=5.21935791 podStartE2EDuration="27.816415351s" totalImagesPullingTime="22.597057441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:55.816344164 +0000 UTC m=+27.758966950" watchObservedRunningTime="2026-07-07 18:50:55.816415351 +0000 UTC m=+27.759038138" Jul 07 18:50:56.614885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.614849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:56.614885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.614876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:56.615175 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.614854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:56.615175 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:56.614959 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:56.615175 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:56.615042 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:56.615175 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:56.615125 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:56.800020 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.799986 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="08d33064b1922c7e2848ca09830d84be8166ac8be6fe02c9ab9eadecaf37ada0" exitCode=0 Jul 07 18:50:56.800550 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.800075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"08d33064b1922c7e2848ca09830d84be8166ac8be6fe02c9ab9eadecaf37ada0"} Jul 07 18:50:56.803531 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:56.803465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"95fcd87593b1fe45466bab8ba6760c50e9e8b0dcdc387789c99a3dbdfa67177e"} Jul 07 18:50:58.616301 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.615999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:50:58.616301 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.616061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:50:58.616855 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:58.616318 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:50:58.616855 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.616087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:50:58.616855 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:58.616391 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:50:58.616855 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:50:58.616452 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:50:58.810547 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.810508 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="48921b15fd03ef43016240cac2ee8fbff8f8377abe532f2a9ffc248c90aa22c6" exitCode=0 Jul 07 18:50:58.810752 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.810590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"48921b15fd03ef43016240cac2ee8fbff8f8377abe532f2a9ffc248c90aa22c6"} Jul 07 18:50:58.814270 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.814237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" event={"ID":"e59d0e33-9a13-4d3a-9f93-eb637735d72f","Type":"ContainerStarted","Data":"3dc902b919b00a9224f112b9e14292719839f09523e203404ca2bc51a98c0510"} Jul 07 18:50:58.814497 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.814481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:58.814573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.814541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:58.814573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.814571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:58.832110 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.832077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:58.832283 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.832243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:50:58.861037 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:50:58.860984 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" podStartSLOduration=9.3770739 podStartE2EDuration="30.860969192s" totalImagesPullingTime="21.483895292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:58.859765106 +0000 UTC m=+30.802387893" watchObservedRunningTime="2026-07-07 18:50:58.860969192 +0000 UTC m=+30.803591979" Jul 07 18:51:00.615108 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:00.615071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:00.615569 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:00.615080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:00.615569 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:00.615189 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:00.615569 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:00.615248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:00.615569 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:00.615319 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:00.615569 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:00.615382 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:02.296862 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.296833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.296871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.296899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297000 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297003 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297023 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297049 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret podName:41b38953-6cb3-4893-ac18-51c6f63c818d nodeName:}" failed. No retries permitted until 2026-07-07 18:51:34.297043019 +0000 UTC m=+66.239665785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret") pod "global-pull-secret-syncer-t68ch" (UID: "41b38953-6cb3-4893-ac18-51c6f63c818d") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297063 2564 projected.go:196] Error preparing data for projected volume kube-api-access-9k6lp for pod openshift-network-diagnostics/network-check-target-2fk95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs podName:0381d901-b528-4f36-98b7-b82be8f1b698 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:34.29707014 +0000 UTC m=+66.239692909 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs") pod "network-metrics-daemon-679vf" (UID: "0381d901-b528-4f36-98b7-b82be8f1b698") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:51:02.297292 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.297113 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp podName:85e42413-b0ac-4c13-89fd-99df22c40ab5 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:34.297097462 +0000 UTC m=+66.239720245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k6lp" (UniqueName: "kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp") pod "network-check-target-2fk95" (UID: "85e42413-b0ac-4c13-89fd-99df22c40ab5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:51:02.614843 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.614746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:02.614843 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.614749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:02.615048 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.614875 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:02.615048 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:02.614749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:02.615048 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.614962 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:02.615196 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:02.615056 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:04.614455 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:04.614418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:04.614885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:04.614417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:04.614885 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:04.614542 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:04.614885 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:04.614584 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:04.614885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:04.614418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:04.614885 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:04.614696 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:06.615109 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:06.615068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:06.615634 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:06.615068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:06.615634 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:06.615214 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:06.615634 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:06.615295 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:06.615634 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:06.615069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:06.615634 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:06.615365 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:07.837266 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:07.837234 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="26831af06ffa0b8ec3805b18466c30e4c86d945fd15749dec90f04c5b8fa1587" exitCode=0 Jul 07 18:51:07.837673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:07.837290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"26831af06ffa0b8ec3805b18466c30e4c86d945fd15749dec90f04c5b8fa1587"} Jul 07 18:51:08.615254 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.615220 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:08.615432 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.615326 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:08.615432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.615335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:08.615432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.615364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:08.615597 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.615431 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:08.615597 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.615514 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:08.843119 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.842930 2564 generic.go:356] "Generic (PLEG): container finished" podID="01107935-02af-4824-a87a-2d9052e0e239" containerID="2eba828721e78da031eabdd09aa70aa6f679cc4ed8379e9558d060eb96cd1016" exitCode=0 Jul 07 18:51:08.843550 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.843011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerDied","Data":"2eba828721e78da031eabdd09aa70aa6f679cc4ed8379e9558d060eb96cd1016"} Jul 07 18:51:08.884494 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.884233 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fk95"] Jul 07 18:51:08.884815 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.884798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:08.885043 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.885017 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:08.887009 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.886634 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-679vf"] Jul 07 18:51:08.887009 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.886749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:08.887009 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.886854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:08.897602 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.897564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t68ch"] Jul 07 18:51:08.898404 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:08.897724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:08.898404 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:08.897825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:09.849168 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:09.849130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69rjl" event={"ID":"01107935-02af-4824-a87a-2d9052e0e239","Type":"ContainerStarted","Data":"4464eabf950050d4a6230215b524e54f7307f8ad3b6dcdbf4f09104d3b54277a"} Jul 07 18:51:09.873105 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:09.873050 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-69rjl" podStartSLOduration=10.296284531 podStartE2EDuration="41.873026995s" totalImagesPullingTime="31.576742464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:09.872537788 +0000 UTC m=+41.815160574" watchObservedRunningTime="2026-07-07 18:51:09.873026995 +0000 UTC m=+41.815649782" Jul 07 18:51:10.620275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:10.620249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:10.620437 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:10.620336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:10.620437 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:10.620343 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:10.620437 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:10.620363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:10.620437 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:10.620428 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:10.620581 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:10.620488 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:12.615234 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.615204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:12.615635 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.615204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:12.615635 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:12.615312 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t68ch" podUID="41b38953-6cb3-4893-ac18-51c6f63c818d" Jul 07 18:51:12.615635 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.615204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:12.615635 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:12.615393 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-679vf" podUID="0381d901-b528-4f36-98b7-b82be8f1b698" Jul 07 18:51:12.615635 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:12.615514 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fk95" podUID="85e42413-b0ac-4c13-89fd-99df22c40ab5" Jul 07 18:51:12.854972 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.854941 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeReady" Jul 07 18:51:12.855151 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.855092 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 18:51:12.908303 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.908222 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6m8j8"] Jul 07 18:51:12.924956 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.924923 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qznkn"] Jul 07 18:51:12.925119 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.925068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:12.929789 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.929755 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 18:51:12.929789 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.929784 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 18:51:12.929983 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.929813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 18:51:12.929983 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.929863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45jgr\"" Jul 07 18:51:12.946972 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.946943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6m8j8"] Jul 07 18:51:12.946972 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.946982 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qznkn"] Jul 07 18:51:12.947149 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.947058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:12.949193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.949170 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 18:51:12.949357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.949279 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 18:51:12.949468 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.949454 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 18:51:12.949567 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.949552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 18:51:12.949621 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:12.949608 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lh7s7\"" Jul 07 18:51:13.006435 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.006401 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lnv4b"] Jul 07 18:51:13.026353 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.026321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnv4b"] Jul 07 18:51:13.026504 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.026451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.028967 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.028942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 18:51:13.029092 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.029006 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b4nbt\"" Jul 07 18:51:13.036532 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.036512 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 18:51:13.038751 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.038729 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:51:13.058529 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.058499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:51:13.058720 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.058590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.061242 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061213 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 18:51:13.061242 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 18:51:13.061397 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061257 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-flmwv\"" Jul 07 18:51:13.061546 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 18:51:13.061610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 18:51:13.061610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 18:51:13.061978 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 18:51:13.062056 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.061994 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 18:51:13.077385 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996859cd-ffc7-444d-b58f-ba1ec0848c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.077385 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce69cba6-e743-40a2-9d41-a3ea064109a4-cert\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.077607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xc7\" (UniqueName: \"kubernetes.io/projected/ce69cba6-e743-40a2-9d41-a3ea064109a4-kube-api-access-r7xc7\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.077607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996859cd-ffc7-444d-b58f-ba1ec0848c19-data-volume\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.077607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.077607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2j6\" (UniqueName: \"kubernetes.io/projected/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-api-access-8g2j6\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.077607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.077605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996859cd-ffc7-444d-b58f-ba1ec0848c19-crio-socket\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.178821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996859cd-ffc7-444d-b58f-ba1ec0848c19-crio-socket\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.178821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.178821 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996859cd-ffc7-444d-b58f-ba1ec0848c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce69cba6-e743-40a2-9d41-a3ea064109a4-cert\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xc7\" (UniqueName: \"kubernetes.io/projected/ce69cba6-e743-40a2-9d41-a3ea064109a4-kube-api-access-r7xc7\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996859cd-ffc7-444d-b58f-ba1ec0848c19-data-volume\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.178982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179017 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/996859cd-ffc7-444d-b58f-ba1ec0848c19-crio-socket\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179078 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c01a889-98bd-41d1-ad71-6d38f93af32c-tmp-dir\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrxs\" (UniqueName: \"kubernetes.io/projected/0c01a889-98bd-41d1-ad71-6d38f93af32c-kube-api-access-vsrxs\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c01a889-98bd-41d1-ad71-6d38f93af32c-config-volume\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c01a889-98bd-41d1-ad71-6d38f93af32c-metrics-tls\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2j6\" (UniqueName: \"kubernetes.io/projected/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-api-access-8g2j6\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/996859cd-ffc7-444d-b58f-ba1ec0848c19-data-volume\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.179491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sd2c\" (UniqueName: \"kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.179752 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.179724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.182939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.182810 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/996859cd-ffc7-444d-b58f-ba1ec0848c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.183051 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.182845 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce69cba6-e743-40a2-9d41-a3ea064109a4-cert\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.191807 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.191779 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2j6\" (UniqueName: \"kubernetes.io/projected/996859cd-ffc7-444d-b58f-ba1ec0848c19-kube-api-access-8g2j6\") pod \"insights-runtime-extractor-qznkn\" (UID: \"996859cd-ffc7-444d-b58f-ba1ec0848c19\") " pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.192109 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.192086 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xc7\" (UniqueName: \"kubernetes.io/projected/ce69cba6-e743-40a2-9d41-a3ea064109a4-kube-api-access-r7xc7\") pod \"ingress-canary-6m8j8\" (UID: \"ce69cba6-e743-40a2-9d41-a3ea064109a4\") " pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.235075 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.235037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6m8j8" Jul 07 18:51:13.256257 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.256231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qznkn" Jul 07 18:51:13.279985 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.279958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c01a889-98bd-41d1-ad71-6d38f93af32c-tmp-dir\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.280117 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.279990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrxs\" (UniqueName: \"kubernetes.io/projected/0c01a889-98bd-41d1-ad71-6d38f93af32c-kube-api-access-vsrxs\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.280117 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c01a889-98bd-41d1-ad71-6d38f93af32c-config-volume\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.280117 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280117 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c01a889-98bd-41d1-ad71-6d38f93af32c-metrics-tls\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.280311 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sd2c\" (UniqueName: \"kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280311 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280409 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280409 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280409 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280392 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c01a889-98bd-41d1-ad71-6d38f93af32c-tmp-dir\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.280409 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.280781 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.280759 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c01a889-98bd-41d1-ad71-6d38f93af32c-config-volume\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.281747 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.281112 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.281747 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.281537 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.281747 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.281705 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.283013 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.282986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c01a889-98bd-41d1-ad71-6d38f93af32c-metrics-tls\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.283091 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.283062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.283275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.283259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.290976 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.290943 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrxs\" (UniqueName: \"kubernetes.io/projected/0c01a889-98bd-41d1-ad71-6d38f93af32c-kube-api-access-vsrxs\") pod \"dns-default-lnv4b\" (UID: \"0c01a889-98bd-41d1-ad71-6d38f93af32c\") " pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.291075 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.290981 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sd2c\" (UniqueName: \"kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c\") pod \"console-765cdc6fdb-dg5ql\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.335498 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.335466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:13.367607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.367380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:13.486331 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.486299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qznkn"] Jul 07 18:51:13.490122 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:13.490091 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996859cd_ffc7_444d_b58f_ba1ec0848c19.slice/crio-0385898ec69657940ba97b6b84e655f7abc1d890177d33e97e3003111a0b130d WatchSource:0}: Error finding container 0385898ec69657940ba97b6b84e655f7abc1d890177d33e97e3003111a0b130d: Status 404 returned error can't find the container with id 0385898ec69657940ba97b6b84e655f7abc1d890177d33e97e3003111a0b130d Jul 07 18:51:13.500196 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.500170 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6m8j8"] Jul 07 18:51:13.506283 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:13.506244 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce69cba6_e743_40a2_9d41_a3ea064109a4.slice/crio-010d4d234136c0fc01d69e49366a6e258f58589088089d3151c2b19904d00e9b WatchSource:0}: Error finding container 010d4d234136c0fc01d69e49366a6e258f58589088089d3151c2b19904d00e9b: Status 404 returned error can't find the container with id 010d4d234136c0fc01d69e49366a6e258f58589088089d3151c2b19904d00e9b Jul 07 18:51:13.544887 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.544859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnv4b"] Jul 07 18:51:13.550034 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:13.549995 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c01a889_98bd_41d1_ad71_6d38f93af32c.slice/crio-36e8eb0a0c0485b45b0e2c9ed1a4dea05f145a67ad6cd834ed61101fd278395b WatchSource:0}: Error finding container 36e8eb0a0c0485b45b0e2c9ed1a4dea05f145a67ad6cd834ed61101fd278395b: Status 404 returned error can't find the container with id 36e8eb0a0c0485b45b0e2c9ed1a4dea05f145a67ad6cd834ed61101fd278395b Jul 07 18:51:13.558718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.558685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:51:13.564376 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:13.564354 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8511035d_2896_4fd6_8ddf_126af7b26416.slice/crio-4482edbc4ff8622a67092d29c271371ebbc232a9d18ea5d69fa236491cba05a8 WatchSource:0}: Error finding container 4482edbc4ff8622a67092d29c271371ebbc232a9d18ea5d69fa236491cba05a8: Status 404 returned error can't find the container with id 4482edbc4ff8622a67092d29c271371ebbc232a9d18ea5d69fa236491cba05a8 Jul 07 18:51:13.858861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.858823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnv4b" event={"ID":"0c01a889-98bd-41d1-ad71-6d38f93af32c","Type":"ContainerStarted","Data":"36e8eb0a0c0485b45b0e2c9ed1a4dea05f145a67ad6cd834ed61101fd278395b"} Jul 07 18:51:13.859909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.859881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qznkn" event={"ID":"996859cd-ffc7-444d-b58f-ba1ec0848c19","Type":"ContainerStarted","Data":"0385898ec69657940ba97b6b84e655f7abc1d890177d33e97e3003111a0b130d"} Jul 07 18:51:13.860932 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.860911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cdc6fdb-dg5ql" event={"ID":"8511035d-2896-4fd6-8ddf-126af7b26416","Type":"ContainerStarted","Data":"4482edbc4ff8622a67092d29c271371ebbc232a9d18ea5d69fa236491cba05a8"} Jul 07 18:51:13.861918 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:13.861901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6m8j8" event={"ID":"ce69cba6-e743-40a2-9d41-a3ea064109a4","Type":"ContainerStarted","Data":"010d4d234136c0fc01d69e49366a6e258f58589088089d3151c2b19904d00e9b"} Jul 07 18:51:14.621576 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.621545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:14.622600 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.622007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:14.622600 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.622430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:14.626743 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.626267 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 18:51:14.629755 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.628610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 18:51:14.629755 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.628893 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 18:51:14.629755 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.629093 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cxlw6\"" Jul 07 18:51:14.629755 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.629310 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 18:51:14.629755 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:14.629549 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l98hl\"" Jul 07 18:51:17.252132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.252096 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m"] Jul 07 18:51:17.291784 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.291747 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m"] Jul 07 18:51:17.291784 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.291793 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl"] Jul 07 18:51:17.291979 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.291894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tzfkn\"" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294392 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294425 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 18:51:17.294694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.294507 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 18:51:17.308028 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.308000 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p77l6"] Jul 07 18:51:17.308219 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.308138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.310507 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.310483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 18:51:17.310619 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.310590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kq548\"" Jul 07 18:51:17.310788 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.310761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 18:51:17.310788 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.310763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 18:51:17.323074 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.323050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl"] Jul 07 18:51:17.323220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.323193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.326274 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.326239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 18:51:17.326399 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.326341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwjgx\"" Jul 07 18:51:17.326399 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.326378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 18:51:17.326504 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.326490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 18:51:17.417575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.417766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.417766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.417766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0ee559-996b-42f8-b67b-da710033b854-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.417766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-metrics-client-ca\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.417931 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-sys\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.417931 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.417931 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-wtmp\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.417931 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpgvp\" (UniqueName: \"kubernetes.io/projected/3d0ee559-996b-42f8-b67b-da710033b854-kube-api-access-bpgvp\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.417931 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-textfile\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkqj\" (UniqueName: \"kubernetes.io/projected/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-kube-api-access-ghkqj\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.417974 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-accelerators-collector-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f274a00c-eb68-44c9-bf3f-938b9b894ccb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jhn\" (UniqueName: \"kubernetes.io/projected/f274a00c-eb68-44c9-bf3f-938b9b894ccb-kube-api-access-w2jhn\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.418152 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.418361 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.418361 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-root\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.418361 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.418211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.518693 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-sys\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518693 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.518693 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-wtmp\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpgvp\" (UniqueName: \"kubernetes.io/projected/3d0ee559-996b-42f8-b67b-da710033b854-kube-api-access-bpgvp\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-sys\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-textfile\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkqj\" (UniqueName: \"kubernetes.io/projected/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-kube-api-access-ghkqj\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-accelerators-collector-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518857 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-wtmp\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f274a00c-eb68-44c9-bf3f-938b9b894ccb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.518933 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jhn\" (UniqueName: \"kubernetes.io/projected/f274a00c-eb68-44c9-bf3f-938b9b894ccb-kube-api-access-w2jhn\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.518996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-root\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519028 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-textfile\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0ee559-996b-42f8-b67b-da710033b854-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.519268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-metrics-client-ca\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.519536 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519470 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f274a00c-eb68-44c9-bf3f-938b9b894ccb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.519536 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519513 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-root\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:17.519474 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:17.519776 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls podName:e0b6036c-d6c9-4aab-9eaf-aa9acb046424 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:18.019758249 +0000 UTC m=+49.962381014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls") pod "node-exporter-p77l6" (UID: "e0b6036c-d6c9-4aab-9eaf-aa9acb046424") : secret "node-exporter-tls" not found Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.519821 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-metrics-client-ca\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:17.519906 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:17.519951 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls podName:3d0ee559-996b-42f8-b67b-da710033b854 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:18.019936409 +0000 UTC m=+49.962559177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-fwtpl" (UID: "3d0ee559-996b-42f8-b67b-da710033b854") : secret "kube-state-metrics-tls" not found Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.520136 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.520179 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-accelerators-collector-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.520223 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.520219 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ee559-996b-42f8-b67b-da710033b854-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.520582 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.520390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0ee559-996b-42f8-b67b-da710033b854-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.522486 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.522465 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.522616 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.522541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.522710 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.522640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.523128 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.523111 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f274a00c-eb68-44c9-bf3f-938b9b894ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.531993 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.531967 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkqj\" (UniqueName: \"kubernetes.io/projected/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-kube-api-access-ghkqj\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:17.532417 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.532402 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpgvp\" (UniqueName: \"kubernetes.io/projected/3d0ee559-996b-42f8-b67b-da710033b854-kube-api-access-bpgvp\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:17.533002 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.532961 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jhn\" (UniqueName: \"kubernetes.io/projected/f274a00c-eb68-44c9-bf3f-938b9b894ccb-kube-api-access-w2jhn\") pod \"openshift-state-metrics-54cf96dfcf-pdj8m\" (UID: \"f274a00c-eb68-44c9-bf3f-938b9b894ccb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:17.603016 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:17.602976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" Jul 07 18:51:18.023357 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.023327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:18.023548 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.023431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:18.026010 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.025987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0ee559-996b-42f8-b67b-da710033b854-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fwtpl\" (UID: \"3d0ee559-996b-42f8-b67b-da710033b854\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:18.026141 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.026051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0b6036c-d6c9-4aab-9eaf-aa9acb046424-node-exporter-tls\") pod \"node-exporter-p77l6\" (UID: \"e0b6036c-d6c9-4aab-9eaf-aa9acb046424\") " pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:18.217224 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.217174 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" Jul 07 18:51:18.232515 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.232482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p77l6" Jul 07 18:51:18.427260 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.427225 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:51:18.443406 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.443380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454254 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wschz\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454298 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454304 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 18:51:18.454477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 18:51:18.454880 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454502 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 18:51:18.454880 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:51:18.454969 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.454956 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 18:51:18.455801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.455788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 18:51:18.629841 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.629841 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgl4\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.629841 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.629992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.630012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630049 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.630029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630238 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.630056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630238 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.630071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.630238 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.630103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731072 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731237 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731237 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731237 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731237 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731237 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.731467 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgl4\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732453 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.732812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.732810 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.733452 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.731984 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.734978 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.734951 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.735412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.735494 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.735568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.735842 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.736438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.737114 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737495 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.737469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.737879 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.737860 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.741899 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.741878 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgl4\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4\") pod \"alertmanager-main-0\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:18.752809 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:18.752780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:51:20.411444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.411411 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t"] Jul 07 18:51:20.441459 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.441435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.444705 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.444681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 18:51:20.444823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.444774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e1r0pn0v7ifqq\"" Jul 07 18:51:20.445751 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.445724 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 18:51:20.445935 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.445810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 18:51:20.446061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.446012 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zbfh9\"" Jul 07 18:51:20.446061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.446048 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 18:51:20.446215 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.446199 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 18:51:20.447837 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.447811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t"] Jul 07 18:51:20.549485 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-grpc-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549673 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222770a6-57c4-4ca6-b75f-e508227e5945-metrics-client-ca\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549786 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgn5\" (UniqueName: \"kubernetes.io/projected/222770a6-57c4-4ca6-b75f-e508227e5945-kube-api-access-mhgn5\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549786 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549786 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.549894 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.549771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650166 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-grpc-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222770a6-57c4-4ca6-b75f-e508227e5945-metrics-client-ca\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgn5\" (UniqueName: \"kubernetes.io/projected/222770a6-57c4-4ca6-b75f-e508227e5945-kube-api-access-mhgn5\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.650428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.650408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.651132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.651077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222770a6-57c4-4ca6-b75f-e508227e5945-metrics-client-ca\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.653565 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.653534 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-grpc-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.653693 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.653535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.653960 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.653934 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.654062 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.654042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.654131 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.654060 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-tls\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.654220 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.654203 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/222770a6-57c4-4ca6-b75f-e508227e5945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.660467 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.660441 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgn5\" (UniqueName: \"kubernetes.io/projected/222770a6-57c4-4ca6-b75f-e508227e5945-kube-api-access-mhgn5\") pod \"thanos-querier-5dcb8865f7-fqk6t\" (UID: \"222770a6-57c4-4ca6-b75f-e508227e5945\") " pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.751780 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.751688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:20.839260 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.839226 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:20.841448 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.841426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.850028 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.850002 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 18:51:20.854467 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.854441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:20.953340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxrp\" (UniqueName: \"kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:20.953577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:20.953475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054155 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054338 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054338 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054338 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxrp\" (UniqueName: \"kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.054575 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.054398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.055210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.055188 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.055331 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.055305 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.055823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.055799 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.055907 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.055889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.058593 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.058412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.061288 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.061267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.066406 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.066380 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxrp\" (UniqueName: \"kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp\") pod \"console-64f74c875d-dsj84\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.158438 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.157756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:21.239548 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.238328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m"] Jul 07 18:51:21.249853 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:21.249571 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf274a00c_eb68_44c9_bf3f_938b9b894ccb.slice/crio-c329fae695a41d28db00cd1a57fb36a1f4fdd375925fa8619f10b542d503995f WatchSource:0}: Error finding container c329fae695a41d28db00cd1a57fb36a1f4fdd375925fa8619f10b542d503995f: Status 404 returned error can't find the container with id c329fae695a41d28db00cd1a57fb36a1f4fdd375925fa8619f10b542d503995f Jul 07 18:51:21.277535 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.272395 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:51:21.277535 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.275289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl"] Jul 07 18:51:21.295363 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.295010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t"] Jul 07 18:51:21.308836 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:21.308746 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222770a6_57c4_4ca6_b75f_e508227e5945.slice/crio-e7f2e99cfc7cd30ca590a003ba85bf68f9d1ee09e1d7bba56125d2cca5eeb4db WatchSource:0}: Error finding container e7f2e99cfc7cd30ca590a003ba85bf68f9d1ee09e1d7bba56125d2cca5eeb4db: Status 404 returned error can't find the container with id e7f2e99cfc7cd30ca590a003ba85bf68f9d1ee09e1d7bba56125d2cca5eeb4db Jul 07 18:51:21.416310 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.416072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:21.420409 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:21.420381 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8687e2b3_0225_4e19_9cda_d1a64290e04d.slice/crio-34f905d2853a05dde145163a6c1e0aa43e5d8859ca964a4f66de1df44e7328dc WatchSource:0}: Error finding container 34f905d2853a05dde145163a6c1e0aa43e5d8859ca964a4f66de1df44e7328dc: Status 404 returned error can't find the container with id 34f905d2853a05dde145163a6c1e0aa43e5d8859ca964a4f66de1df44e7328dc Jul 07 18:51:21.582815 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.582728 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57bcf444-whfc9"] Jul 07 18:51:21.585418 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.585398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.588362 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.587825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 18:51:21.588362 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.587851 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 18:51:21.588362 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.587883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2rlaolbnd8h8s\"" Jul 07 18:51:21.588362 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.587825 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 18:51:21.588362 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.588171 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 18:51:21.588633 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.588614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nfdgp\"" Jul 07 18:51:21.600125 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.600098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bcf444-whfc9"] Jul 07 18:51:21.669662 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-tls\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.669823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-metrics-server-audit-profiles\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.669823 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.669939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-client-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.669939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-client-certs\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.669939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdvh\" (UniqueName: \"kubernetes.io/projected/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-kube-api-access-srdvh\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.670093 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.669990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-audit-log\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770666 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-audit-log\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-tls\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-metrics-server-audit-profiles\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-client-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770993 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-client-certs\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.770993 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.770908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdvh\" (UniqueName: \"kubernetes.io/projected/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-kube-api-access-srdvh\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.771106 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.771083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-audit-log\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.772041 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.771988 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.772156 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.772094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-metrics-server-audit-profiles\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.773720 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.773697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-tls\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.773820 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.773733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-client-ca-bundle\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.773891 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.773868 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-secret-metrics-server-client-certs\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.779909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.779859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdvh\" (UniqueName: \"kubernetes.io/projected/ce0d8dca-f01f-49a3-aca1-75ea52ae578e-kube-api-access-srdvh\") pod \"metrics-server-57bcf444-whfc9\" (UID: \"ce0d8dca-f01f-49a3-aca1-75ea52ae578e\") " pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.890171 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.890069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qznkn" event={"ID":"996859cd-ffc7-444d-b58f-ba1ec0848c19","Type":"ContainerStarted","Data":"0357412bbbbd4e42d904eca06691aed25396c0d57da5bdd1732d5972494efa72"} Jul 07 18:51:21.891882 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.891853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cdc6fdb-dg5ql" event={"ID":"8511035d-2896-4fd6-8ddf-126af7b26416","Type":"ContainerStarted","Data":"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1"} Jul 07 18:51:21.893955 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.893513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6m8j8" event={"ID":"ce69cba6-e743-40a2-9d41-a3ea064109a4","Type":"ContainerStarted","Data":"ef03aca7f549c68feb37d5418e795fdb0ef13ae5fbb9c4e4fcd3ba4b865f4319"} Jul 07 18:51:21.895986 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.895754 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:21.896305 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.896274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"e7f2e99cfc7cd30ca590a003ba85bf68f9d1ee09e1d7bba56125d2cca5eeb4db"} Jul 07 18:51:21.899870 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.899771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" event={"ID":"f274a00c-eb68-44c9-bf3f-938b9b894ccb","Type":"ContainerStarted","Data":"d0c7f4aae762ffe13a0e262fb35d194f32616b878aa4dd7c3324e8b4156b53d2"} Jul 07 18:51:21.899870 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.899820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" event={"ID":"f274a00c-eb68-44c9-bf3f-938b9b894ccb","Type":"ContainerStarted","Data":"8c32120210f20ec4ea123e7464e9d1cb0cf4b7cdf3a9550912d19a61e62871f0"} Jul 07 18:51:21.899870 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.899837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" event={"ID":"f274a00c-eb68-44c9-bf3f-938b9b894ccb","Type":"ContainerStarted","Data":"c329fae695a41d28db00cd1a57fb36a1f4fdd375925fa8619f10b542d503995f"} Jul 07 18:51:21.901303 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.901279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p77l6" event={"ID":"e0b6036c-d6c9-4aab-9eaf-aa9acb046424","Type":"ContainerStarted","Data":"c0f79e710ac62efce73104b5ca4bb8b8542f93e7e3b7e045d92d1b18e704a7ed"} Jul 07 18:51:21.902613 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.902588 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"115b18898c2e481852ced0c76d27d604df8827f9fb8904f84e704577d105558c"} Jul 07 18:51:21.903935 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.903907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" event={"ID":"3d0ee559-996b-42f8-b67b-da710033b854","Type":"ContainerStarted","Data":"6b4e647e857dfb694eb7ecf0e2ed58970f11968f64918f63883d2e7fc409913f"} Jul 07 18:51:21.906999 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.906975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnv4b" event={"ID":"0c01a889-98bd-41d1-ad71-6d38f93af32c","Type":"ContainerStarted","Data":"1eef029d47a99a0b066f453d96b01309fab93556de3bfc846f658a48fae5b235"} Jul 07 18:51:21.907096 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.907021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnv4b" event={"ID":"0c01a889-98bd-41d1-ad71-6d38f93af32c","Type":"ContainerStarted","Data":"ade3a7471e6272b230217812580226050280699cdd6dc294d7377eacd282463c"} Jul 07 18:51:21.907173 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.907158 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:21.911901 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.911853 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765cdc6fdb-dg5ql" podStartSLOduration=1.418186792 podStartE2EDuration="8.911835456s" totalImagesPullingTime="7.493648664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:21.911300074 +0000 UTC m=+53.853922848" watchObservedRunningTime="2026-07-07 18:51:21.911835456 +0000 UTC m=+53.854458245" Jul 07 18:51:21.913525 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.912980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f74c875d-dsj84" event={"ID":"8687e2b3-0225-4e19-9cda-d1a64290e04d","Type":"ContainerStarted","Data":"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff"} Jul 07 18:51:21.913525 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.913021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f74c875d-dsj84" event={"ID":"8687e2b3-0225-4e19-9cda-d1a64290e04d","Type":"ContainerStarted","Data":"34f905d2853a05dde145163a6c1e0aa43e5d8859ca964a4f66de1df44e7328dc"} Jul 07 18:51:21.934473 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.933531 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6m8j8" podStartSLOduration=2.785036451 podStartE2EDuration="9.933505643s" totalImagesPullingTime="7.148469192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:21.930895588 +0000 UTC m=+53.873518376" watchObservedRunningTime="2026-07-07 18:51:21.933505643 +0000 UTC m=+53.876128436" Jul 07 18:51:21.957611 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:21.956793 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lnv4b" podStartSLOduration=2.48129714 podStartE2EDuration="9.956769076s" totalImagesPullingTime="7.475471936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:21.955497864 +0000 UTC m=+53.898120651" watchObservedRunningTime="2026-07-07 18:51:21.956769076 +0000 UTC m=+53.899391845" Jul 07 18:51:22.047871 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.046200 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64f74c875d-dsj84" podStartSLOduration=2.046177177 podStartE2EDuration="2.046177177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:21.978282828 +0000 UTC m=+53.920905630" watchObservedRunningTime="2026-07-07 18:51:22.046177177 +0000 UTC m=+53.988799948" Jul 07 18:51:22.047871 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.046523 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn"] Jul 07 18:51:22.050058 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.050029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.053043 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.053016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 18:51:22.053043 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.053035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nf5jz\"" Jul 07 18:51:22.057832 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.057804 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn"] Jul 07 18:51:22.078201 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.078167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pc6\" (UniqueName: \"kubernetes.io/projected/364310fa-0994-4ed4-942d-3eff39767124-kube-api-access-88pc6\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.078354 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.078218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.178998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.178918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pc6\" (UniqueName: \"kubernetes.io/projected/364310fa-0994-4ed4-942d-3eff39767124-kube-api-access-88pc6\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.178998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.178974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.179203 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:22.179183 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 18:51:22.179255 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:22.179247 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert podName:364310fa-0994-4ed4-942d-3eff39767124 nodeName:}" failed. No retries permitted until 2026-07-07 18:51:22.679227152 +0000 UTC m=+54.621849941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-fmgdn" (UID: "364310fa-0994-4ed4-942d-3eff39767124") : secret "monitoring-plugin-cert" not found Jul 07 18:51:22.191692 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.191613 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pc6\" (UniqueName: \"kubernetes.io/projected/364310fa-0994-4ed4-942d-3eff39767124-kube-api-access-88pc6\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.472235 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.472151 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc"] Jul 07 18:51:22.478097 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.478067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.480630 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.480604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 18:51:22.481073 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.480936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 18:51:22.481073 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.480984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 18:51:22.481229 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.480934 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 18:51:22.483183 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.481760 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 18:51:22.483183 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.482325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ct8q2\"" Jul 07 18:51:22.488929 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.488902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc"] Jul 07 18:51:22.489199 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.489179 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 18:51:22.583559 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583559 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-serving-certs-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-federate-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frspx\" (UniqueName: \"kubernetes.io/projected/508d25dd-bef1-42e8-8216-bf5df9538300-kube-api-access-frspx\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-metrics-client-ca\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.583996 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.583900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685234 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-metrics-client-ca\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-serving-certs-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-federate-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685493 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.685947 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.685514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frspx\" (UniqueName: \"kubernetes.io/projected/508d25dd-bef1-42e8-8216-bf5df9538300-kube-api-access-frspx\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.686464 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.686068 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-serving-certs-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.686464 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.686140 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-metrics-client-ca\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.686464 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.686326 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.688259 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.688229 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.688588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.688568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364310fa-0994-4ed4-942d-3eff39767124-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-fmgdn\" (UID: \"364310fa-0994-4ed4-942d-3eff39767124\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:22.688718 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.688595 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-secret-telemeter-client\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.688866 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.688847 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-telemeter-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.688989 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.688974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/508d25dd-bef1-42e8-8216-bf5df9538300-federate-client-tls\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.694334 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.694308 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frspx\" (UniqueName: \"kubernetes.io/projected/508d25dd-bef1-42e8-8216-bf5df9538300-kube-api-access-frspx\") pod \"telemeter-client-64fb65ff5f-ln2bc\" (UID: \"508d25dd-bef1-42e8-8216-bf5df9538300\") " pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.794465 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.794416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" Jul 07 18:51:22.967197 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:22.967161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:23.082026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.081994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bcf444-whfc9"] Jul 07 18:51:23.177421 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:23.177379 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0d8dca_f01f_49a3_aca1_75ea52ae578e.slice/crio-d32c9dbe19c758d7c46bdf50b9a7ec932221030ccbe528bfa8e93c0c9942bea0 WatchSource:0}: Error finding container d32c9dbe19c758d7c46bdf50b9a7ec932221030ccbe528bfa8e93c0c9942bea0: Status 404 returned error can't find the container with id d32c9dbe19c758d7c46bdf50b9a7ec932221030ccbe528bfa8e93c0c9942bea0 Jul 07 18:51:23.369518 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.369078 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:23.369518 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.369455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:23.378817 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.378783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:23.379716 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.379626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc"] Jul 07 18:51:23.394687 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.394479 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn"] Jul 07 18:51:23.408585 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:23.408540 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508d25dd_bef1_42e8_8216_bf5df9538300.slice/crio-b5cfdcbb99ea3a99a2325301e4fb360b17fc049a039990c74eb01a1726a0f4f8 WatchSource:0}: Error finding container b5cfdcbb99ea3a99a2325301e4fb360b17fc049a039990c74eb01a1726a0f4f8: Status 404 returned error can't find the container with id b5cfdcbb99ea3a99a2325301e4fb360b17fc049a039990c74eb01a1726a0f4f8 Jul 07 18:51:23.497674 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.496219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:51:23.503899 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.503860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.506359 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.506334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-67x5c\"" Jul 07 18:51:23.506359 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.506381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 18:51:23.506607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.506338 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 18:51:23.507792 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.507642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 18:51:23.508187 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.508137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.508471 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.508609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.508770 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.508939 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509014 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509180 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509255 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509260 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 18:51:23.509606 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.509419 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2cgpv3kh1megh\"" Jul 07 18:51:23.510624 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.510604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 18:51:23.516108 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.516039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:51:23.598813 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.598813 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.598813 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.598999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjtq\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.599423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.599408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700314 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.700706 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjtq\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701022 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.700921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701594 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.701568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.701773 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.701624 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.702170 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.702134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.702276 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.702221 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.703570 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.702525 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.703570 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.703165 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.706425 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.705577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.706425 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.705885 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708036 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708003 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708247 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708268 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708258 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708476 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708452 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708631 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708611 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708711 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708678 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708789 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708767 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.708908 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.708885 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.710026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.709975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.710413 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.710391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.710577 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.710560 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.717798 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.717762 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjtq\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq\") pod \"prometheus-k8s-0\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.820008 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.819724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:23.924053 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.923970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qznkn" event={"ID":"996859cd-ffc7-444d-b58f-ba1ec0848c19","Type":"ContainerStarted","Data":"d999cdf01734f86bd4a777c70b50ea4ab52e07820c31c07f42c34c960585a10c"} Jul 07 18:51:23.926499 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.926462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" event={"ID":"f274a00c-eb68-44c9-bf3f-938b9b894ccb","Type":"ContainerStarted","Data":"5c33bb9efd412fb2f35c113a9ad915f19d45ae5b2e09ed68789e9897881cb512"} Jul 07 18:51:23.928360 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.928330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p77l6" event={"ID":"e0b6036c-d6c9-4aab-9eaf-aa9acb046424","Type":"ContainerStarted","Data":"811ae1efc5dab768528b8cb923e7bac93812422fd13779ca6487a2194f7f6b8d"} Jul 07 18:51:23.929740 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.929712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" event={"ID":"ce0d8dca-f01f-49a3-aca1-75ea52ae578e","Type":"ContainerStarted","Data":"d32c9dbe19c758d7c46bdf50b9a7ec932221030ccbe528bfa8e93c0c9942bea0"} Jul 07 18:51:23.931440 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.931416 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d" exitCode=0 Jul 07 18:51:23.931547 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.931499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d"} Jul 07 18:51:23.933347 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.933322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" event={"ID":"3d0ee559-996b-42f8-b67b-da710033b854","Type":"ContainerStarted","Data":"3362c376b9c5f6579957b39d39987a0c128a528a5769db992709165ad38ec13d"} Jul 07 18:51:23.933394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.933357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" event={"ID":"3d0ee559-996b-42f8-b67b-da710033b854","Type":"ContainerStarted","Data":"86c4865bce9b1d410ebd8af5d6408d50305d0b9d3af3fd75c31291fa3531feb2"} Jul 07 18:51:23.934599 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.934576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" event={"ID":"508d25dd-bef1-42e8-8216-bf5df9538300","Type":"ContainerStarted","Data":"b5cfdcbb99ea3a99a2325301e4fb360b17fc049a039990c74eb01a1726a0f4f8"} Jul 07 18:51:23.939672 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.939599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:51:23.945063 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:23.945018 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pdj8m" podStartSLOduration=5.182863799 podStartE2EDuration="6.945003087s" totalImagesPullingTime="1.762139288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:23.943916531 +0000 UTC m=+55.886539323" watchObservedRunningTime="2026-07-07 18:51:23.945003087 +0000 UTC m=+55.887625876" Jul 07 18:51:23.990344 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:23.990309 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364310fa_0994_4ed4_942d_3eff39767124.slice/crio-ec6680bd8ab5fbb7e74d9f04cb47b426882a97b32b15f88b05c7dfa308cc6ca9 WatchSource:0}: Error finding container ec6680bd8ab5fbb7e74d9f04cb47b426882a97b32b15f88b05c7dfa308cc6ca9: Status 404 returned error can't find the container with id ec6680bd8ab5fbb7e74d9f04cb47b426882a97b32b15f88b05c7dfa308cc6ca9 Jul 07 18:51:24.167105 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.167077 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:51:24.175174 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:24.175083 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e5c303_29e6_4748_8626_1ddf8c2d10dc.slice/crio-ad1283b5e42ac29b937393b1cd3b3b40cc850f141b8ce7867e3649a64454df2c WatchSource:0}: Error finding container ad1283b5e42ac29b937393b1cd3b3b40cc850f141b8ce7867e3649a64454df2c: Status 404 returned error can't find the container with id ad1283b5e42ac29b937393b1cd3b3b40cc850f141b8ce7867e3649a64454df2c Jul 07 18:51:24.946376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.946183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"658018143f43006ecf7e9ec9468f3b774fabbbf44d7eb0f1cb33f377dfbb3605"} Jul 07 18:51:24.946376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.946279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"17220e5229e98029cb57f46b38abab10aa43e49bcff3461b7da09baef2a75f81"} Jul 07 18:51:24.946376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.946322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"dbd279deba8fcf65cbc8dc0f15d088d34950b3573bb85ba6a32c0058e36d3bf9"} Jul 07 18:51:24.952029 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.950944 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0b6036c-d6c9-4aab-9eaf-aa9acb046424" containerID="811ae1efc5dab768528b8cb923e7bac93812422fd13779ca6487a2194f7f6b8d" exitCode=0 Jul 07 18:51:24.952029 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.951032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p77l6" event={"ID":"e0b6036c-d6c9-4aab-9eaf-aa9acb046424","Type":"ContainerDied","Data":"811ae1efc5dab768528b8cb923e7bac93812422fd13779ca6487a2194f7f6b8d"} Jul 07 18:51:24.957424 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.955485 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167" exitCode=0 Jul 07 18:51:24.957424 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.955586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167"} Jul 07 18:51:24.957424 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.955670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"ad1283b5e42ac29b937393b1cd3b3b40cc850f141b8ce7867e3649a64454df2c"} Jul 07 18:51:24.965397 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.964633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" event={"ID":"3d0ee559-996b-42f8-b67b-da710033b854","Type":"ContainerStarted","Data":"98b1411272ebb8b5e38e13f14337dda572ade3f8d407747f5c9b71a85ae66ce5"} Jul 07 18:51:24.967877 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:24.967776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" event={"ID":"364310fa-0994-4ed4-942d-3eff39767124","Type":"ContainerStarted","Data":"ec6680bd8ab5fbb7e74d9f04cb47b426882a97b32b15f88b05c7dfa308cc6ca9"} Jul 07 18:51:25.028154 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:25.027468 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fwtpl" podStartSLOduration=6.124797771 podStartE2EDuration="8.027443147s" totalImagesPullingTime="1.902645376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:24.995543222 +0000 UTC m=+56.938166010" watchObservedRunningTime="2026-07-07 18:51:25.027443147 +0000 UTC m=+56.970065937" Jul 07 18:51:25.459296 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:25.459255 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:30.830508 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.830480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g75c6" Jul 07 18:51:30.994406 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.994364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"ddcd7862753cc3df9b3396766b307826cfba45c9049607c18c31110f639af932"} Jul 07 18:51:30.994566 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.994440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"7639d9212f042fb1bfcb6ae5a93551d450932f85e6e065d7a10bba51c37a8827"} Jul 07 18:51:30.994566 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.994497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" event={"ID":"222770a6-57c4-4ca6-b75f-e508227e5945","Type":"ContainerStarted","Data":"5f1910f589b7dedae7f1620d5255122bec969549e7de6f67f34168e15e33b6d3"} Jul 07 18:51:30.994566 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.994557 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:30.996887 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.996860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p77l6" event={"ID":"e0b6036c-d6c9-4aab-9eaf-aa9acb046424","Type":"ContainerStarted","Data":"3daf25066aef03cd393923e2db4f7354017655db7efe02fa2f4d55c179da592b"} Jul 07 18:51:30.997018 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:30.996921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p77l6" event={"ID":"e0b6036c-d6c9-4aab-9eaf-aa9acb046424","Type":"ContainerStarted","Data":"dfdb353bd7d43154af47c206381eec8be42859bad106e009b806054d888c3807"} Jul 07 18:51:31.004156 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.004127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" event={"ID":"ce0d8dca-f01f-49a3-aca1-75ea52ae578e","Type":"ContainerStarted","Data":"fe14729cfa6dd2ed01a8ec0dfb8f95ef873384d30f347d9575dd42d64bbe3cee"} Jul 07 18:51:31.006428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.006405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" Jul 07 18:51:31.007807 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.007778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3"} Jul 07 18:51:31.007918 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.007851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738"} Jul 07 18:51:31.007918 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.007902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9"} Jul 07 18:51:31.007999 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.007940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d"} Jul 07 18:51:31.007999 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.007976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346"} Jul 07 18:51:31.008076 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.008006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerStarted","Data":"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3"} Jul 07 18:51:31.011071 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4"} Jul 07 18:51:31.011167 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89"} Jul 07 18:51:31.011167 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79"} Jul 07 18:51:31.011289 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8"} Jul 07 18:51:31.011289 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa"} Jul 07 18:51:31.011289 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.011286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerStarted","Data":"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179"} Jul 07 18:51:31.013192 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.013170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" event={"ID":"508d25dd-bef1-42e8-8216-bf5df9538300","Type":"ContainerStarted","Data":"8e985169969948d294046d511140fa5e9fc405e29b5278eb55080f975f32f31e"} Jul 07 18:51:31.013292 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.013225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" event={"ID":"508d25dd-bef1-42e8-8216-bf5df9538300","Type":"ContainerStarted","Data":"cfd407a3e1a4cd0689959c3a4d0397082f9d0c384ed61421378775362d4baa8e"} Jul 07 18:51:31.013292 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.013254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" event={"ID":"508d25dd-bef1-42e8-8216-bf5df9538300","Type":"ContainerStarted","Data":"0133fd4131f6d2fda98ed2596e5497b11bb232bebf40ecf9471fc6d50e6ba3d0"} Jul 07 18:51:31.014624 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.014600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" event={"ID":"364310fa-0994-4ed4-942d-3eff39767124","Type":"ContainerStarted","Data":"301c494f43825e30ce290ee7540d6ca5ff64542cf546756ab63fadec11d85aba"} Jul 07 18:51:31.014772 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.014754 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:31.017615 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.017045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qznkn" event={"ID":"996859cd-ffc7-444d-b58f-ba1ec0848c19","Type":"ContainerStarted","Data":"3156cc4d18d63d7c54b1960c3c1d5a53a27f48ef6ad19617d24fe90f604f94c4"} Jul 07 18:51:31.018292 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.018257 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dcb8865f7-fqk6t" podStartSLOduration=2.577458506 podStartE2EDuration="11.018240855s" totalImagesPullingTime="8.440782349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.017626411 +0000 UTC m=+62.960249199" watchObservedRunningTime="2026-07-07 18:51:31.018240855 +0000 UTC m=+62.960863636" Jul 07 18:51:31.020978 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.020954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" Jul 07 18:51:31.070137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.070080 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p77l6" podStartSLOduration=11.832585787 podStartE2EDuration="14.070063621s" totalImagesPullingTime="2.115996015s" totalInitContainerRuntime="121.481819ms" isStatefulPod=true podCreationTimestamp="2026-07-07 18:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.067810235 +0000 UTC m=+63.010433023" watchObservedRunningTime="2026-07-07 18:51:31.070063621 +0000 UTC m=+63.012686406" Jul 07 18:51:31.087765 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.087633 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qznkn" podStartSLOduration=2.8120119260000003 podStartE2EDuration="19.087610717s" totalImagesPullingTime="16.275598791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:51:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.086416158 +0000 UTC m=+63.029038949" watchObservedRunningTime="2026-07-07 18:51:31.087610717 +0000 UTC m=+63.030233505" Jul 07 18:51:31.116239 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.116191 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.063731112 podStartE2EDuration="13.116175939s" totalImagesPullingTime="8.024961496s" totalInitContainerRuntime="27.483331ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.11411497 +0000 UTC m=+63.056737752" watchObservedRunningTime="2026-07-07 18:51:31.116175939 +0000 UTC m=+63.058798725" Jul 07 18:51:31.133288 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.133217 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" podStartSLOduration=4.352696183 podStartE2EDuration="10.133190674s" totalImagesPullingTime="5.780494491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.131325211 +0000 UTC m=+63.073947998" watchObservedRunningTime="2026-07-07 18:51:31.133190674 +0000 UTC m=+63.075813462" Jul 07 18:51:31.149297 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.149247 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-fmgdn" podStartSLOduration=4.178961694 podStartE2EDuration="9.149229797s" totalImagesPullingTime="4.970268103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.148115497 +0000 UTC m=+63.090738298" watchObservedRunningTime="2026-07-07 18:51:31.149229797 +0000 UTC m=+63.091852632" Jul 07 18:51:31.158482 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.158451 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:31.168895 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.168831 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64fb65ff5f-ln2bc" podStartSLOduration=3.615035097 podStartE2EDuration="9.168812178s" totalImagesPullingTime="5.553777081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.167535632 +0000 UTC m=+63.110158418" watchObservedRunningTime="2026-07-07 18:51:31.168812178 +0000 UTC m=+63.111434967" Jul 07 18:51:31.197793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.197748 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.060278446 podStartE2EDuration="8.19773038s" totalImagesPullingTime="5.112620758s" totalInitContainerRuntime="24.831176ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:31.195155739 +0000 UTC m=+63.137778526" watchObservedRunningTime="2026-07-07 18:51:31.19773038 +0000 UTC m=+63.140353166" Jul 07 18:51:31.919536 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.919504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lnv4b" Jul 07 18:51:31.961431 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.961396 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:51:31.966958 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.966925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:31.972169 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:31.972142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:51:32.099374 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.099339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.100467 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.100434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4sd\" (UniqueName: \"kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.100593 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.100509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.100692 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.100640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.101076 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.101053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.101233 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.101216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.101466 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.101440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4sd\" (UniqueName: \"kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202963 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202963 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202963 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.202963 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.202944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.203137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.203010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.203421 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.203391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.203548 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.203396 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.203670 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.203634 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.203750 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.203733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.205416 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.205391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.205494 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.205429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.211668 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.211625 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4sd\" (UniqueName: \"kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd\") pod \"console-54fb97cd9-bfxdb\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.279300 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.279256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:32.408542 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:32.408323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:51:32.410721 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:32.410638 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03faef26_3245_4083_9546_6d2b4657358c.slice/crio-c765080f45581d9492242478d978c89049ba393a5db7c25d211ee8ae7dded136 WatchSource:0}: Error finding container c765080f45581d9492242478d978c89049ba393a5db7c25d211ee8ae7dded136: Status 404 returned error can't find the container with id c765080f45581d9492242478d978c89049ba393a5db7c25d211ee8ae7dded136 Jul 07 18:51:33.029472 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:33.029431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fb97cd9-bfxdb" event={"ID":"03faef26-3245-4083-9546-6d2b4657358c","Type":"ContainerStarted","Data":"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3"} Jul 07 18:51:33.029942 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:33.029484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fb97cd9-bfxdb" event={"ID":"03faef26-3245-4083-9546-6d2b4657358c","Type":"ContainerStarted","Data":"c765080f45581d9492242478d978c89049ba393a5db7c25d211ee8ae7dded136"} Jul 07 18:51:33.050140 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:33.050085 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54fb97cd9-bfxdb" podStartSLOduration=2.050069148 podStartE2EDuration="2.050069148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:33.048400746 +0000 UTC m=+64.991023534" watchObservedRunningTime="2026-07-07 18:51:33.050069148 +0000 UTC m=+64.992691934" Jul 07 18:51:33.820239 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:33.820201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:51:34.324702 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.324661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:34.325114 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.324727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:34.325114 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.324813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:34.329121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.329098 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 18:51:34.329255 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.329098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 18:51:34.329856 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.329842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 18:51:34.337928 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.337869 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41b38953-6cb3-4893-ac18-51c6f63c818d-original-pull-secret\") pod \"global-pull-secret-syncer-t68ch\" (UID: \"41b38953-6cb3-4893-ac18-51c6f63c818d\") " pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:34.338051 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.337933 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0381d901-b528-4f36-98b7-b82be8f1b698-metrics-certs\") pod \"network-metrics-daemon-679vf\" (UID: \"0381d901-b528-4f36-98b7-b82be8f1b698\") " pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:34.338213 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.338194 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 18:51:34.348688 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.348640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6lp\" (UniqueName: \"kubernetes.io/projected/85e42413-b0ac-4c13-89fd-99df22c40ab5-kube-api-access-9k6lp\") pod \"network-check-target-2fk95\" (UID: \"85e42413-b0ac-4c13-89fd-99df22c40ab5\") " pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:34.452440 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.452405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cxlw6\"" Jul 07 18:51:34.454153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.454139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l98hl\"" Jul 07 18:51:34.457927 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.457909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t68ch" Jul 07 18:51:34.460124 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.460105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:34.462761 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.462747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-679vf" Jul 07 18:51:34.606010 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.605976 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t68ch"] Jul 07 18:51:34.609271 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:34.609239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b38953_6cb3_4893_ac18_51c6f63c818d.slice/crio-f7d197ab08df05a6c09178d929127cf55900f149533f7110c4359d4397eebf31 WatchSource:0}: Error finding container f7d197ab08df05a6c09178d929127cf55900f149533f7110c4359d4397eebf31: Status 404 returned error can't find the container with id f7d197ab08df05a6c09178d929127cf55900f149533f7110c4359d4397eebf31 Jul 07 18:51:34.848952 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.848927 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fk95"] Jul 07 18:51:34.850805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:34.850777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-679vf"] Jul 07 18:51:34.850955 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:34.850935 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0381d901_b528_4f36_98b7_b82be8f1b698.slice/crio-708da5716fe34048adae968888248fcd741774478e330c13976fec093c304e6e WatchSource:0}: Error finding container 708da5716fe34048adae968888248fcd741774478e330c13976fec093c304e6e: Status 404 returned error can't find the container with id 708da5716fe34048adae968888248fcd741774478e330c13976fec093c304e6e Jul 07 18:51:34.852290 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:51:34.852256 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e42413_b0ac_4c13_89fd_99df22c40ab5.slice/crio-dab4c4299d0313057122b50ae4b0dc29766466679c2a929bec9528078b4224a8 WatchSource:0}: Error finding container dab4c4299d0313057122b50ae4b0dc29766466679c2a929bec9528078b4224a8: Status 404 returned error can't find the container with id dab4c4299d0313057122b50ae4b0dc29766466679c2a929bec9528078b4224a8 Jul 07 18:51:35.039829 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:35.039798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t68ch" event={"ID":"41b38953-6cb3-4893-ac18-51c6f63c818d","Type":"ContainerStarted","Data":"f7d197ab08df05a6c09178d929127cf55900f149533f7110c4359d4397eebf31"} Jul 07 18:51:35.041018 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:35.040989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-679vf" event={"ID":"0381d901-b528-4f36-98b7-b82be8f1b698","Type":"ContainerStarted","Data":"708da5716fe34048adae968888248fcd741774478e330c13976fec093c304e6e"} Jul 07 18:51:35.042130 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:35.042105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fk95" event={"ID":"85e42413-b0ac-4c13-89fd-99df22c40ab5","Type":"ContainerStarted","Data":"dab4c4299d0313057122b50ae4b0dc29766466679c2a929bec9528078b4224a8"} Jul 07 18:51:36.053836 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:36.053804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-679vf" event={"ID":"0381d901-b528-4f36-98b7-b82be8f1b698","Type":"ContainerStarted","Data":"88389f994d3a40306e18f4b2bed4c5bad6c8ab6077adcafaea31895ef6d97cda"} Jul 07 18:51:37.063052 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:37.063012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-679vf" event={"ID":"0381d901-b528-4f36-98b7-b82be8f1b698","Type":"ContainerStarted","Data":"b1c662475a0bf3fe2f2d3ce0a1bc288039bde4f13da4c095ba23ce7a5cf4eedf"} Jul 07 18:51:40.080073 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.080034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t68ch" event={"ID":"41b38953-6cb3-4893-ac18-51c6f63c818d","Type":"ContainerStarted","Data":"4d9e6b73e2d829a9213b93e1ef22a09f12126251bcc87bbee6e3e41a97d6093d"} Jul 07 18:51:40.081705 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.081681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fk95" event={"ID":"85e42413-b0ac-4c13-89fd-99df22c40ab5","Type":"ContainerStarted","Data":"7f827289817fd1df106c353009ec06c70bac107c6cde766fe54a870f806cd45a"} Jul 07 18:51:40.081820 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.081802 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:51:40.095698 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.095578 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-679vf" podStartSLOduration=71.039220291 podStartE2EDuration="1m12.095557426s" totalImagesPullingTime="1.056337135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:37.079526132 +0000 UTC m=+69.022148922" watchObservedRunningTime="2026-07-07 18:51:40.095557426 +0000 UTC m=+72.038180214" Jul 07 18:51:40.095914 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.095883 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t68ch" podStartSLOduration=65.747675816 podStartE2EDuration="1m11.095870456s" totalImagesPullingTime="5.34819464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:40.095642849 +0000 UTC m=+72.038265637" watchObservedRunningTime="2026-07-07 18:51:40.095870456 +0000 UTC m=+72.038493243" Jul 07 18:51:40.112626 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:40.112565 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2fk95" podStartSLOduration=67.011700205 podStartE2EDuration="1m12.112547349s" totalImagesPullingTime="5.100847144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:51:40.111232909 +0000 UTC m=+72.053855698" watchObservedRunningTime="2026-07-07 18:51:40.112547349 +0000 UTC m=+72.055170135" Jul 07 18:51:41.896435 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:41.896391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:41.896934 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:41.896506 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:51:42.279915 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:42.279882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:42.280121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:42.279973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:42.284801 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:42.284780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:43.097902 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:43.097877 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:51:43.144553 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:43.144518 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:51:50.489366 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.489211 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64f74c875d-dsj84" podUID="8687e2b3-0225-4e19-9cda-d1a64290e04d" containerName="console" containerID="cri-o://68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff" gracePeriod=15 Jul 07 18:51:50.776527 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.776490 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:50.788971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.788932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.788971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.788979 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789180 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789002 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dxrp\" (UniqueName: \"kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789180 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789034 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789180 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789060 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789180 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789097 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789180 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789138 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config\") pod \"8687e2b3-0225-4e19-9cda-d1a64290e04d\" (UID: \"8687e2b3-0225-4e19-9cda-d1a64290e04d\") " Jul 07 18:51:50.789413 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789367 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca" (OuterVolumeSpecName: "service-ca") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:51:50.789494 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789468 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:51:50.789576 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789551 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.789633 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789594 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.789633 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789591 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config" (OuterVolumeSpecName: "console-config") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:51:50.789747 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.789673 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:51:50.791806 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.791777 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:51:50.791920 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.791829 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp" (OuterVolumeSpecName: "kube-api-access-7dxrp") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "kube-api-access-7dxrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:51:50.791920 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.791844 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8687e2b3-0225-4e19-9cda-d1a64290e04d" (UID: "8687e2b3-0225-4e19-9cda-d1a64290e04d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:51:50.890376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.890338 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dxrp\" (UniqueName: \"kubernetes.io/projected/8687e2b3-0225-4e19-9cda-d1a64290e04d-kube-api-access-7dxrp\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.890376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.890370 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.890376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.890379 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.890376 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.890387 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:50.890637 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:50.890397 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8687e2b3-0225-4e19-9cda-d1a64290e04d-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:51:51.127442 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.127408 2564 generic.go:356] "Generic (PLEG): container finished" podID="8687e2b3-0225-4e19-9cda-d1a64290e04d" containerID="68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff" exitCode=0 Jul 07 18:51:51.127612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.127451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f74c875d-dsj84" event={"ID":"8687e2b3-0225-4e19-9cda-d1a64290e04d","Type":"ContainerDied","Data":"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff"} Jul 07 18:51:51.127612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.127489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f74c875d-dsj84" event={"ID":"8687e2b3-0225-4e19-9cda-d1a64290e04d","Type":"ContainerDied","Data":"34f905d2853a05dde145163a6c1e0aa43e5d8859ca964a4f66de1df44e7328dc"} Jul 07 18:51:51.127612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.127509 2564 scope.go:117] "RemoveContainer" containerID="68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff" Jul 07 18:51:51.127612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.127455 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f74c875d-dsj84" Jul 07 18:51:51.137045 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.137027 2564 scope.go:117] "RemoveContainer" containerID="68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff" Jul 07 18:51:51.137366 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:51:51.137346 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff\": container with ID starting with 68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff not found: ID does not exist" containerID="68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff" Jul 07 18:51:51.137448 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.137375 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff"} err="failed to get container status \"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff\": rpc error: code = NotFound desc = could not find container \"68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff\": container with ID starting with 68253d06fe705fbf278ca3e9ff3ff38d51000ad719062e65898b05d0c333faff not found: ID does not exist" Jul 07 18:51:51.149098 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.149061 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:51.155111 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:51.155082 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64f74c875d-dsj84"] Jul 07 18:51:52.619707 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:51:52.619673 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8687e2b3-0225-4e19-9cda-d1a64290e04d" path="/var/lib/kubelet/pods/8687e2b3-0225-4e19-9cda-d1a64290e04d/volumes" Jul 07 18:52:01.902017 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:01.901985 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:52:01.906468 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:01.906435 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57bcf444-whfc9" Jul 07 18:52:08.165875 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.165830 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-765cdc6fdb-dg5ql" podUID="8511035d-2896-4fd6-8ddf-126af7b26416" containerName="console" containerID="cri-o://1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1" gracePeriod=15 Jul 07 18:52:08.422881 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.422816 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:52:08.555948 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.555901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.555983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556010 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556069 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sd2c\" (UniqueName: \"kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556132 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556114 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config\") pod \"8511035d-2896-4fd6-8ddf-126af7b26416\" (UID: \"8511035d-2896-4fd6-8ddf-126af7b26416\") " Jul 07 18:52:08.556388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556331 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:08.556442 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556419 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:08.556793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556699 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config" (OuterVolumeSpecName: "console-config") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:08.556793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.556718 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca" (OuterVolumeSpecName: "service-ca") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:08.558449 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.558408 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:08.558449 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.558436 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c" (OuterVolumeSpecName: "kube-api-access-7sd2c") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "kube-api-access-7sd2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:08.558614 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.558462 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8511035d-2896-4fd6-8ddf-126af7b26416" (UID: "8511035d-2896-4fd6-8ddf-126af7b26416"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:08.657121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.657080 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sd2c\" (UniqueName: \"kubernetes.io/projected/8511035d-2896-4fd6-8ddf-126af7b26416-kube-api-access-7sd2c\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:08.657121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.657113 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:08.657121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.657123 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8511035d-2896-4fd6-8ddf-126af7b26416-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:08.657121 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.657130 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:08.657390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:08.657138 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8511035d-2896-4fd6-8ddf-126af7b26416-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:09.202410 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.202373 2564 generic.go:356] "Generic (PLEG): container finished" podID="8511035d-2896-4fd6-8ddf-126af7b26416" containerID="1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1" exitCode=0 Jul 07 18:52:09.202825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.202419 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cdc6fdb-dg5ql" Jul 07 18:52:09.202825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.202459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cdc6fdb-dg5ql" event={"ID":"8511035d-2896-4fd6-8ddf-126af7b26416","Type":"ContainerDied","Data":"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1"} Jul 07 18:52:09.202825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.202514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cdc6fdb-dg5ql" event={"ID":"8511035d-2896-4fd6-8ddf-126af7b26416","Type":"ContainerDied","Data":"4482edbc4ff8622a67092d29c271371ebbc232a9d18ea5d69fa236491cba05a8"} Jul 07 18:52:09.202825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.202538 2564 scope.go:117] "RemoveContainer" containerID="1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1" Jul 07 18:52:09.211048 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.211025 2564 scope.go:117] "RemoveContainer" containerID="1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1" Jul 07 18:52:09.211327 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:09.211307 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1\": container with ID starting with 1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1 not found: ID does not exist" containerID="1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1" Jul 07 18:52:09.211401 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.211335 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1"} err="failed to get container status \"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1\": rpc error: code = NotFound desc = could not find container \"1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1\": container with ID starting with 1f3d7e46e33688caa47712fdb694d3c067dcd735e81af8aa93e5a10ba5f4e7f1 not found: ID does not exist" Jul 07 18:52:09.220101 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.220063 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:52:09.224756 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:09.224728 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765cdc6fdb-dg5ql"] Jul 07 18:52:10.627906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:10.627839 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8511035d-2896-4fd6-8ddf-126af7b26416" path="/var/lib/kubelet/pods/8511035d-2896-4fd6-8ddf-126af7b26416/volumes" Jul 07 18:52:11.087936 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:11.087907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2fk95" Jul 07 18:52:25.094988 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:25.093442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:25.120149 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:25.120121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:25.292127 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:25.292103 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:37.638742 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.638707 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.638989 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="alertmanager" containerID="cri-o://88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179" gracePeriod=120 Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.639075 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-metric" containerID="cri-o://f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89" gracePeriod=120 Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.639101 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-web" containerID="cri-o://18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8" gracePeriod=120 Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.639116 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="prom-label-proxy" containerID="cri-o://a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4" gracePeriod=120 Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.639135 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy" containerID="cri-o://621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79" gracePeriod=120 Jul 07 18:52:37.639275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:37.639153 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="config-reloader" containerID="cri-o://3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa" gracePeriod=120 Jul 07 18:52:38.332422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332389 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4" exitCode=0 Jul 07 18:52:38.332422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332416 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89" exitCode=0 Jul 07 18:52:38.332422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332423 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79" exitCode=0 Jul 07 18:52:38.332422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332430 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa" exitCode=0 Jul 07 18:52:38.332422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332438 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179" exitCode=0 Jul 07 18:52:38.332766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4"} Jul 07 18:52:38.332766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89"} Jul 07 18:52:38.332766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79"} Jul 07 18:52:38.332766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa"} Jul 07 18:52:38.332766 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.332620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179"} Jul 07 18:52:38.884364 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.884338 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:38.925521 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925487 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925612 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mgl4\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.925731 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925740 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925760 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.926038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.925928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets\") pod \"b199428f-8403-47ef-b82d-98a9d5c08705\" (UID: \"b199428f-8403-47ef-b82d-98a9d5c08705\") " Jul 07 18:52:38.927190 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.926709 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:38.927824 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.927778 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:38.927941 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.927828 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:38.929458 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.929415 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.929570 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.929490 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:38.930529 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.930489 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.931212 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.931177 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4" (OuterVolumeSpecName: "kube-api-access-7mgl4") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "kube-api-access-7mgl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:38.931484 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.931450 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.931804 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.931777 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.931922 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.931898 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out" (OuterVolumeSpecName: "config-out") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:38.932186 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.932167 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume" (OuterVolumeSpecName: "config-volume") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.937130 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.937104 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:38.945851 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:38.945824 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config" (OuterVolumeSpecName: "web-config") pod "b199428f-8403-47ef-b82d-98a9d5c08705" (UID: "b199428f-8403-47ef-b82d-98a9d5c08705"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:39.026889 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026862 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-cluster-tls-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.026889 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026891 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-main-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026901 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026910 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgl4\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-kube-api-access-7mgl4\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026918 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-main-db\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026927 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026935 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026943 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026951 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b199428f-8403-47ef-b82d-98a9d5c08705-config-out\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026959 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026968 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199428f-8403-47ef-b82d-98a9d5c08705-metrics-client-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.026996 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b199428f-8403-47ef-b82d-98a9d5c08705-web-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.027061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.027004 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b199428f-8403-47ef-b82d-98a9d5c08705-tls-assets\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:39.339091 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.339011 2564 generic.go:356] "Generic (PLEG): container finished" podID="b199428f-8403-47ef-b82d-98a9d5c08705" containerID="18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8" exitCode=0 Jul 07 18:52:39.339091 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.339086 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.339271 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.339096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8"} Jul 07 18:52:39.339271 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.339166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b199428f-8403-47ef-b82d-98a9d5c08705","Type":"ContainerDied","Data":"115b18898c2e481852ced0c76d27d604df8827f9fb8904f84e704577d105558c"} Jul 07 18:52:39.339271 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.339202 2564 scope.go:117] "RemoveContainer" containerID="a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4" Jul 07 18:52:39.346847 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.346828 2564 scope.go:117] "RemoveContainer" containerID="f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89" Jul 07 18:52:39.354283 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.354261 2564 scope.go:117] "RemoveContainer" containerID="621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79" Jul 07 18:52:39.361523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.361503 2564 scope.go:117] "RemoveContainer" containerID="18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8" Jul 07 18:52:39.364317 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.364293 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:39.368462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.368438 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:39.370438 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.370422 2564 scope.go:117] "RemoveContainer" containerID="3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa" Jul 07 18:52:39.377969 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.377944 2564 scope.go:117] "RemoveContainer" containerID="88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179" Jul 07 18:52:39.385423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.385409 2564 scope.go:117] "RemoveContainer" containerID="6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d" Jul 07 18:52:39.394339 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.394292 2564 scope.go:117] "RemoveContainer" containerID="a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4" Jul 07 18:52:39.394897 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.394872 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4\": container with ID starting with a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4 not found: ID does not exist" containerID="a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4" Jul 07 18:52:39.395116 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.394932 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4"} err="failed to get container status \"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4\": rpc error: code = NotFound desc = could not find container \"a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4\": container with ID starting with a031f9925be8d28c7362da9fc84df2cdb3576c0c26110ecd20700e8a131de7a4 not found: ID does not exist" Jul 07 18:52:39.395116 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.394963 2564 scope.go:117] "RemoveContainer" containerID="f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89" Jul 07 18:52:39.396268 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.396248 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89\": container with ID starting with f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89 not found: ID does not exist" containerID="f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89" Jul 07 18:52:39.396340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396274 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89"} err="failed to get container status \"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89\": rpc error: code = NotFound desc = could not find container \"f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89\": container with ID starting with f8b6539f658bbaafd508887f61a01963a71b03cf5d7ace90eb840d87fac47d89 not found: ID does not exist" Jul 07 18:52:39.396340 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396291 2564 scope.go:117] "RemoveContainer" containerID="621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79" Jul 07 18:52:39.396592 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.396572 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79\": container with ID starting with 621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79 not found: ID does not exist" containerID="621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79" Jul 07 18:52:39.396643 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396602 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79"} err="failed to get container status \"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79\": rpc error: code = NotFound desc = could not find container \"621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79\": container with ID starting with 621289d9915a0a7eacbecd3f105a76e0181f96f25e9e324374fd164f89684b79 not found: ID does not exist" Jul 07 18:52:39.396643 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396619 2564 scope.go:117] "RemoveContainer" containerID="18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8" Jul 07 18:52:39.396869 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.396852 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8\": container with ID starting with 18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8 not found: ID does not exist" containerID="18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8" Jul 07 18:52:39.396906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396873 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8"} err="failed to get container status \"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8\": rpc error: code = NotFound desc = could not find container \"18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8\": container with ID starting with 18990e93e54e73ca01ef43cc2fa59b1083b0327337043eef3ab32c570ccec6a8 not found: ID does not exist" Jul 07 18:52:39.396906 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.396888 2564 scope.go:117] "RemoveContainer" containerID="3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa" Jul 07 18:52:39.397112 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.397096 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa\": container with ID starting with 3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa not found: ID does not exist" containerID="3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa" Jul 07 18:52:39.397165 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397114 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa"} err="failed to get container status \"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa\": rpc error: code = NotFound desc = could not find container \"3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa\": container with ID starting with 3b0145f04f32a0084b876ae003444fc3fd91cd8bc37a6aa2dfc99cd81d0ce0fa not found: ID does not exist" Jul 07 18:52:39.397165 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397129 2564 scope.go:117] "RemoveContainer" containerID="88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179" Jul 07 18:52:39.397365 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.397347 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179\": container with ID starting with 88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179 not found: ID does not exist" containerID="88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179" Jul 07 18:52:39.397428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397369 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179"} err="failed to get container status \"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179\": rpc error: code = NotFound desc = could not find container \"88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179\": container with ID starting with 88242ee237a8529f6c44ede40b07d9948d14215643ce9cf846f2b12fa9a7b179 not found: ID does not exist" Jul 07 18:52:39.397428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397382 2564 scope.go:117] "RemoveContainer" containerID="6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d" Jul 07 18:52:39.397505 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397430 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:39.397623 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:39.397600 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d\": container with ID starting with 6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d not found: ID does not exist" containerID="6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d" Jul 07 18:52:39.397681 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397627 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d"} err="failed to get container status \"6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d\": rpc error: code = NotFound desc = could not find container \"6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d\": container with ID starting with 6072edba9b15bcfd6c170db8f0c742f5bb5e7afd38981a3e228a4c71760f392d not found: ID does not exist" Jul 07 18:52:39.397799 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397785 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="alertmanager" Jul 07 18:52:39.397835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397801 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="alertmanager" Jul 07 18:52:39.397835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397816 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy" Jul 07 18:52:39.397835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397821 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy" Jul 07 18:52:39.397835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397828 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8511035d-2896-4fd6-8ddf-126af7b26416" containerName="console" Jul 07 18:52:39.397835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397833 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8511035d-2896-4fd6-8ddf-126af7b26416" containerName="console" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397841 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8687e2b3-0225-4e19-9cda-d1a64290e04d" containerName="console" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397846 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8687e2b3-0225-4e19-9cda-d1a64290e04d" containerName="console" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397859 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="config-reloader" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397865 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="config-reloader" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397875 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="init-config-reloader" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397881 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="init-config-reloader" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397888 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-metric" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397893 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-metric" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397901 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="prom-label-proxy" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397905 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="prom-label-proxy" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-web" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397921 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-web" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397964 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8687e2b3-0225-4e19-9cda-d1a64290e04d" containerName="console" Jul 07 18:52:39.397971 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397971 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="prom-label-proxy" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397977 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-metric" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397984 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="alertmanager" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397990 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8511035d-2896-4fd6-8ddf-126af7b26416" containerName="console" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.397996 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="config-reloader" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.398001 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy" Jul 07 18:52:39.398330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.398007 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" containerName="kube-rbac-proxy-web" Jul 07 18:52:39.407099 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.407065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.409872 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.409852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 18:52:39.410057 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 18:52:39.410146 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 18:52:39.410146 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 18:52:39.410146 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 18:52:39.410373 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 18:52:39.410373 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410368 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 18:52:39.410487 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410368 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wschz\"" Jul 07 18:52:39.410487 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.410354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 18:52:39.416314 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.416286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 18:52:39.417158 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.417139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:39.430628 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-volume\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430628 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-out\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430800 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430800 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430800 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqr6\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-kube-api-access-dwqr6\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.430909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.431026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.431026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.430994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.431084 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.431026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.431084 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.431044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-web-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.531978 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.531942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-web-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-volume\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-out\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532193 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqr6\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-kube-api-access-dwqr6\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532619 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532619 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532619 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.532797 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.532722 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.533394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.533326 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.533394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.533374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6560098a-03e8-4fb8-b784-b48cc4f2d746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.535758 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535688 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.535758 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535691 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-volume\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.535758 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535686 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.535758 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535742 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6560098a-03e8-4fb8-b784-b48cc4f2d746-config-out\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.536026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.536026 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.535966 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-web-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.536154 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.536134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.536212 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.536190 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.537038 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.537022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6560098a-03e8-4fb8-b784-b48cc4f2d746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.541690 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.541667 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqr6\" (UniqueName: \"kubernetes.io/projected/6560098a-03e8-4fb8-b784-b48cc4f2d746-kube-api-access-dwqr6\") pod \"alertmanager-main-0\" (UID: \"6560098a-03e8-4fb8-b784-b48cc4f2d746\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.718628 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.718537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 18:52:39.855540 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:39.855506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 18:52:39.859298 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:52:39.859250 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6560098a_03e8_4fb8_b784_b48cc4f2d746.slice/crio-d2188f36b50d887c4958cc7ffb96baa3753c839e90c075dbe70233d6d71fef1e WatchSource:0}: Error finding container d2188f36b50d887c4958cc7ffb96baa3753c839e90c075dbe70233d6d71fef1e: Status 404 returned error can't find the container with id d2188f36b50d887c4958cc7ffb96baa3753c839e90c075dbe70233d6d71fef1e Jul 07 18:52:40.343887 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:40.343853 2564 generic.go:356] "Generic (PLEG): container finished" podID="6560098a-03e8-4fb8-b784-b48cc4f2d746" containerID="62b77eaeee18593ef7f722ca3d5dae29a097f0db9ff7a9c22d98cc264d8691e8" exitCode=0 Jul 07 18:52:40.344356 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:40.343939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerDied","Data":"62b77eaeee18593ef7f722ca3d5dae29a097f0db9ff7a9c22d98cc264d8691e8"} Jul 07 18:52:40.344356 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:40.344017 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"d2188f36b50d887c4958cc7ffb96baa3753c839e90c075dbe70233d6d71fef1e"} Jul 07 18:52:40.623171 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:40.623144 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b199428f-8403-47ef-b82d-98a9d5c08705" path="/var/lib/kubelet/pods/b199428f-8403-47ef-b82d-98a9d5c08705/volumes" Jul 07 18:52:41.352511 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"24d85b64661055200483e7b371ea97cfe1c4c84220b741cf62aeb2812fbb475d"} Jul 07 18:52:41.352939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"3c4f726a171d3e172ac35dbbffdd20ad44ad4c2038f4164b0e887f3b0aa5ea03"} Jul 07 18:52:41.352939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"7e3e0ae50f15de1848efcb8ceed12f1b255600e8d5e20d34241e521a651cba2e"} Jul 07 18:52:41.352939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"aa8bb331f6d8dbbaed961574d0a1c5bcfe6fa11b921f8ddea015310b3fcbcc90"} Jul 07 18:52:41.352939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"87f2551c321a69b5cb0dc8c222b7da5315fe17064558cd57150a8d04fd900233"} Jul 07 18:52:41.352939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.352735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6560098a-03e8-4fb8-b784-b48cc4f2d746","Type":"ContainerStarted","Data":"9037a415e4f1352d15a94f32a6e8ade4145401b4b6061034821b4250ef5d4d5a"} Jul 07 18:52:41.379778 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.379684 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.3643958290000002 podStartE2EDuration="2.379662429s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.2666ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:52:41.378988851 +0000 UTC m=+133.321611637" watchObservedRunningTime="2026-07-07 18:52:41.379662429 +0000 UTC m=+133.322285211" Jul 07 18:52:41.882993 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.882951 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:41.883353 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883318 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="prometheus" containerID="cri-o://7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3" gracePeriod=600 Jul 07 18:52:41.883440 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883344 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3" gracePeriod=600 Jul 07 18:52:41.883440 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883371 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy" containerID="cri-o://c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738" gracePeriod=600 Jul 07 18:52:41.883440 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883324 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="thanos-sidecar" containerID="cri-o://10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d" gracePeriod=600 Jul 07 18:52:41.883582 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883342 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-web" containerID="cri-o://20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9" gracePeriod=600 Jul 07 18:52:41.883640 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:41.883342 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="config-reloader" containerID="cri-o://5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346" gracePeriod=600 Jul 07 18:52:42.360122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360084 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3" exitCode=0 Jul 07 18:52:42.360122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360110 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738" exitCode=0 Jul 07 18:52:42.360122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360117 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d" exitCode=0 Jul 07 18:52:42.360122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360123 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346" exitCode=0 Jul 07 18:52:42.360122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360129 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3" exitCode=0 Jul 07 18:52:42.360605 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3"} Jul 07 18:52:42.360605 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738"} Jul 07 18:52:42.360605 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360267 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d"} Jul 07 18:52:42.360605 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346"} Jul 07 18:52:42.360605 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:42.360341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3"} Jul 07 18:52:43.130423 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.130398 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.165003 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.164951 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165226 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.164997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165226 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165226 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165226 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165207 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165246 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165333 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wjtq\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165404 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165432 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165475 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.165514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165562 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165614 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165707 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs\") pod \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\" (UID: \"67e5c303-29e6-4748-8626-1ddf8c2d10dc\") " Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.165965 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.167336 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-metrics-client-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.167419 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.168186 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.168525 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.168765 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.168813 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.170077 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.169181 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:43.171000 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.170928 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.171054 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.171026 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.171631 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.171590 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.171963 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.171935 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.172105 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.172084 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config" (OuterVolumeSpecName: "config") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.173111 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.173065 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.173200 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.173122 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:43.173335 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.173316 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.173543 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.173523 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out" (OuterVolumeSpecName: "config-out") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:43.174137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.174102 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq" (OuterVolumeSpecName: "kube-api-access-6wjtq") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "kube-api-access-6wjtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:43.175192 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.175162 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.185710 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.185637 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config" (OuterVolumeSpecName: "web-config") pod "67e5c303-29e6-4748-8626-1ddf8c2d10dc" (UID: "67e5c303-29e6-4748-8626-1ddf8c2d10dc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267778 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-kube-rbac-proxy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267812 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config-out\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267825 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267837 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-tls-assets\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267850 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267861 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-web-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.267873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267874 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267888 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-metrics-client-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267900 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267913 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267924 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267936 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267947 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-db\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267958 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267972 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-grpc-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267983 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67e5c303-29e6-4748-8626-1ddf8c2d10dc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.267994 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.268005 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wjtq\" (UniqueName: \"kubernetes.io/projected/67e5c303-29e6-4748-8626-1ddf8c2d10dc-kube-api-access-6wjtq\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.268252 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.268017 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67e5c303-29e6-4748-8626-1ddf8c2d10dc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:52:43.367127 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.367087 2564 generic.go:356] "Generic (PLEG): container finished" podID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerID="20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9" exitCode=0 Jul 07 18:52:43.367522 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.367173 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.367522 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.367170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9"} Jul 07 18:52:43.367522 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.367256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67e5c303-29e6-4748-8626-1ddf8c2d10dc","Type":"ContainerDied","Data":"ad1283b5e42ac29b937393b1cd3b3b40cc850f141b8ce7867e3649a64454df2c"} Jul 07 18:52:43.367522 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.367303 2564 scope.go:117] "RemoveContainer" containerID="b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3" Jul 07 18:52:43.375761 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.375743 2564 scope.go:117] "RemoveContainer" containerID="c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738" Jul 07 18:52:43.382733 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.382716 2564 scope.go:117] "RemoveContainer" containerID="20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9" Jul 07 18:52:43.389467 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.389438 2564 scope.go:117] "RemoveContainer" containerID="10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d" Jul 07 18:52:43.392722 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.392694 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:43.397102 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.396956 2564 scope.go:117] "RemoveContainer" containerID="5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346" Jul 07 18:52:43.399359 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.399337 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:43.404204 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.404188 2564 scope.go:117] "RemoveContainer" containerID="7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3" Jul 07 18:52:43.414201 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.414185 2564 scope.go:117] "RemoveContainer" containerID="59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167" Jul 07 18:52:43.425459 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.425437 2564 scope.go:117] "RemoveContainer" containerID="b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3" Jul 07 18:52:43.425862 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.425842 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3\": container with ID starting with b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3 not found: ID does not exist" containerID="b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3" Jul 07 18:52:43.425937 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.425872 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3"} err="failed to get container status \"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3\": rpc error: code = NotFound desc = could not find container \"b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3\": container with ID starting with b7a1bfe88a492c76d1c3c236a32312031c30d3bc5c9868ca571d246f881db0a3 not found: ID does not exist" Jul 07 18:52:43.425937 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.425891 2564 scope.go:117] "RemoveContainer" containerID="c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738" Jul 07 18:52:43.426213 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.426194 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738\": container with ID starting with c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738 not found: ID does not exist" containerID="c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738" Jul 07 18:52:43.426251 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426219 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738"} err="failed to get container status \"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738\": rpc error: code = NotFound desc = could not find container \"c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738\": container with ID starting with c13a08daf7add49fc42388ef1673b92679f4a7e095c37b291f60343331df3738 not found: ID does not exist" Jul 07 18:52:43.426251 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426236 2564 scope.go:117] "RemoveContainer" containerID="20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9" Jul 07 18:52:43.426359 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426338 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:43.426473 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.426460 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9\": container with ID starting with 20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9 not found: ID does not exist" containerID="20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9" Jul 07 18:52:43.426512 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426478 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9"} err="failed to get container status \"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9\": rpc error: code = NotFound desc = could not find container \"20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9\": container with ID starting with 20bc5ecb3cc80af5b7027bfc1b00b60064e6fd0c06a219aaca0d5bb0720774b9 not found: ID does not exist" Jul 07 18:52:43.426512 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426490 2564 scope.go:117] "RemoveContainer" containerID="10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d" Jul 07 18:52:43.426692 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426675 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="config-reloader" Jul 07 18:52:43.426736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426705 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="config-reloader" Jul 07 18:52:43.426736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426718 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy" Jul 07 18:52:43.426736 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.426715 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d\": container with ID starting with 10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d not found: ID does not exist" containerID="10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d" Jul 07 18:52:43.426736 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426725 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426735 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d"} err="failed to get container status \"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d\": rpc error: code = NotFound desc = could not find container \"10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d\": container with ID starting with 10df2f826cda38e483e8dcf4cd5d46f679bf05940276044bc3c8b9e20871845d not found: ID does not exist" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426746 2564 scope.go:117] "RemoveContainer" containerID="5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426747 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-thanos" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426757 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-thanos" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426768 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="init-config-reloader" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426774 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="init-config-reloader" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426790 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="prometheus" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426794 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="prometheus" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426800 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="thanos-sidecar" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426804 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="thanos-sidecar" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-web" Jul 07 18:52:43.426861 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426816 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-web" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426870 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="thanos-sidecar" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426878 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="prometheus" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426885 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426890 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="config-reloader" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426896 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-web" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.426901 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" containerName="kube-rbac-proxy-thanos" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.426986 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346\": container with ID starting with 5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346 not found: ID does not exist" containerID="5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.427003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346"} err="failed to get container status \"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346\": rpc error: code = NotFound desc = could not find container \"5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346\": container with ID starting with 5643b404875a7ad0c7cbad59137daf57a587703a7b502ef8bea7295b2b605346 not found: ID does not exist" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.427016 2564 scope.go:117] "RemoveContainer" containerID="7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.427229 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3\": container with ID starting with 7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3 not found: ID does not exist" containerID="7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.427246 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3"} err="failed to get container status \"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3\": rpc error: code = NotFound desc = could not find container \"7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3\": container with ID starting with 7aac95abb7a26346f25e0f7daaa283f91f22bf1b4dfdc59cfc6db0b806cce8c3 not found: ID does not exist" Jul 07 18:52:43.427299 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.427259 2564 scope.go:117] "RemoveContainer" containerID="59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167" Jul 07 18:52:43.427720 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:52:43.427473 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167\": container with ID starting with 59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167 not found: ID does not exist" containerID="59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167" Jul 07 18:52:43.427720 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.427488 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167"} err="failed to get container status \"59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167\": rpc error: code = NotFound desc = could not find container \"59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167\": container with ID starting with 59ef4ccff6461435e5b4480da26a3ee3f939128c4e3e5b7ec19fbd0dbad14167 not found: ID does not exist" Jul 07 18:52:43.430751 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.430737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.433444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.433424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 18:52:43.433777 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.433759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 18:52:43.433920 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.433828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 18:52:43.433920 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.433856 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2cgpv3kh1megh\"" Jul 07 18:52:43.434137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.433990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 18:52:43.434137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434048 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 18:52:43.434137 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 18:52:43.434311 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434227 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 18:52:43.434462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 18:52:43.434561 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 18:52:43.434561 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 18:52:43.434909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-67x5c\"" Jul 07 18:52:43.435005 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.434949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 18:52:43.435069 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.435040 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 18:52:43.436229 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.436210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 18:52:43.444014 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.442789 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 18:52:43.447574 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.447553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:43.468983 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.468955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469134 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.468998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469134 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469134 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469134 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469308 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b656s\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-kube-api-access-b656s\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469477 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.469715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.469504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.569924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.569887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.569941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.569965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.569990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570118 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b656s\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-kube-api-access-b656s\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570388 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570694 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570873 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.570975 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.570937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.571043 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.571022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.572210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.571215 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.572210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.571629 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.572210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.571776 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.572210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.571929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.572567 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.572294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.573212 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.573169 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.573331 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.573226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.573711 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.573627 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.574670 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.574223 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.574670 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.574603 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.574835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.574792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.574835 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.574827 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.575010 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.574987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.575092 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.575018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.575179 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.575156 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.575390 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.575365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.575600 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.575583 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.576071 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.576053 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.576261 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.576246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.581061 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.581045 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b656s\" (UniqueName: \"kubernetes.io/projected/dd7e1103-217e-44b6-8ed4-cb72afe4fdf9-kube-api-access-b656s\") pod \"prometheus-k8s-0\" (UID: \"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.740838 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.740782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:43.882306 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:43.882240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 18:52:43.884363 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:52:43.884339 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7e1103_217e_44b6_8ed4_cb72afe4fdf9.slice/crio-b7297986501853942c274c0ac5d3d8a4d02f0d791f5307314eadc6e72a31a52e WatchSource:0}: Error finding container b7297986501853942c274c0ac5d3d8a4d02f0d791f5307314eadc6e72a31a52e: Status 404 returned error can't find the container with id b7297986501853942c274c0ac5d3d8a4d02f0d791f5307314eadc6e72a31a52e Jul 07 18:52:44.373011 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:44.372973 2564 generic.go:356] "Generic (PLEG): container finished" podID="dd7e1103-217e-44b6-8ed4-cb72afe4fdf9" containerID="9c030e3c40a456c11b37ebc81a366222c34d1b3e8b41dcdfda7c06363190abd2" exitCode=0 Jul 07 18:52:44.373422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:44.373067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerDied","Data":"9c030e3c40a456c11b37ebc81a366222c34d1b3e8b41dcdfda7c06363190abd2"} Jul 07 18:52:44.373422 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:44.373139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"b7297986501853942c274c0ac5d3d8a4d02f0d791f5307314eadc6e72a31a52e"} Jul 07 18:52:44.621558 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:44.621411 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e5c303-29e6-4748-8626-1ddf8c2d10dc" path="/var/lib/kubelet/pods/67e5c303-29e6-4748-8626-1ddf8c2d10dc/volumes" Jul 07 18:52:45.382462 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"a19bdf03a0297a31ee4730f4a643b0796711b760f570047cde4f96b1d31f1c89"} Jul 07 18:52:45.382825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"5c77018a485679e0c74d02dd3811902ad9ae885a3200562dafa97a6cf19dec60"} Jul 07 18:52:45.382825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382543 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"ba76d9718a9c96f39c58096cf28e026155cd10b98362db59fdf4561fbedb8f12"} Jul 07 18:52:45.382825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382588 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"13ad70333beab61521e7b97fec01f0691e3c321405555beec7508be80170f544"} Jul 07 18:52:45.382825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"5c343c1d1ba945750d7748b110c0ee39357d814d301cb3818a361bbb00db957d"} Jul 07 18:52:45.382825 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.382676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd7e1103-217e-44b6-8ed4-cb72afe4fdf9","Type":"ContainerStarted","Data":"42de592732457c0e42fc86d629fea6e8d81688ada3a02875e1a2dd8f0f1a3ba0"} Jul 07 18:52:45.410730 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:45.410678 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.39294321 podStartE2EDuration="2.410642104s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.698894ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:52:45.408470907 +0000 UTC m=+137.351093729" watchObservedRunningTime="2026-07-07 18:52:45.410642104 +0000 UTC m=+137.353264890" Jul 07 18:52:48.741172 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:48.741120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:52:48.973370 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:52:48.973333 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:53:13.993812 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:13.993753 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-54fb97cd9-bfxdb" podUID="03faef26-3245-4083-9546-6d2b4657358c" containerName="console" containerID="cri-o://d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3" gracePeriod=15 Jul 07 18:53:14.237724 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.237698 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:53:14.339396 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339410 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339498 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339554 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv4sd\" (UniqueName: \"kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339588 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config\") pod \"03faef26-3245-4083-9546-6d2b4657358c\" (UID: \"03faef26-3245-4083-9546-6d2b4657358c\") " Jul 07 18:53:14.339920 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.339852 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca" (OuterVolumeSpecName: "service-ca") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:53:14.340051 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.340026 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:53:14.340109 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.340040 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:53:14.340109 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.340067 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config" (OuterVolumeSpecName: "console-config") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:53:14.341719 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.341697 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:53:14.341890 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.341871 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd" (OuterVolumeSpecName: "kube-api-access-lv4sd") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "kube-api-access-lv4sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:53:14.341928 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.341896 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "03faef26-3245-4083-9546-6d2b4657358c" (UID: "03faef26-3245-4083-9546-6d2b4657358c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:53:14.440828 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440792 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.440828 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440821 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.440828 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440831 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.440828 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440838 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03faef26-3245-4083-9546-6d2b4657358c-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.441083 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440846 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4sd\" (UniqueName: \"kubernetes.io/projected/03faef26-3245-4083-9546-6d2b4657358c-kube-api-access-lv4sd\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.441083 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440855 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.441083 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.440863 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03faef26-3245-4083-9546-6d2b4657358c-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 18:53:14.509100 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.509063 2564 generic.go:356] "Generic (PLEG): container finished" podID="03faef26-3245-4083-9546-6d2b4657358c" containerID="d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3" exitCode=0 Jul 07 18:53:14.509275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.509112 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fb97cd9-bfxdb" Jul 07 18:53:14.509275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.509152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fb97cd9-bfxdb" event={"ID":"03faef26-3245-4083-9546-6d2b4657358c","Type":"ContainerDied","Data":"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3"} Jul 07 18:53:14.509275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.509217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fb97cd9-bfxdb" event={"ID":"03faef26-3245-4083-9546-6d2b4657358c","Type":"ContainerDied","Data":"c765080f45581d9492242478d978c89049ba393a5db7c25d211ee8ae7dded136"} Jul 07 18:53:14.509275 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.509243 2564 scope.go:117] "RemoveContainer" containerID="d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3" Jul 07 18:53:14.518043 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.518023 2564 scope.go:117] "RemoveContainer" containerID="d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3" Jul 07 18:53:14.518322 ip-10-0-132-43 kubenswrapper[2564]: E0707 18:53:14.518305 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3\": container with ID starting with d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3 not found: ID does not exist" containerID="d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3" Jul 07 18:53:14.518375 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.518336 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3"} err="failed to get container status \"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3\": rpc error: code = NotFound desc = could not find container \"d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3\": container with ID starting with d5fcda3f7d938d9e1ce9d09ddc51e42a89150b3695233e517ecbb223f12535f3 not found: ID does not exist" Jul 07 18:53:14.530377 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.530341 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:53:14.534803 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.534777 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54fb97cd9-bfxdb"] Jul 07 18:53:14.624747 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:14.624668 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03faef26-3245-4083-9546-6d2b4657358c" path="/var/lib/kubelet/pods/03faef26-3245-4083-9546-6d2b4657358c/volumes" Jul 07 18:53:43.741879 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:43.741851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:53:43.756996 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:43.756973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:53:44.640622 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:44.640598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 18:53:55.803343 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.803298 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mfwq"] Jul 07 18:53:55.803805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.803608 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03faef26-3245-4083-9546-6d2b4657358c" containerName="console" Jul 07 18:53:55.803805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.803619 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03faef26-3245-4083-9546-6d2b4657358c" containerName="console" Jul 07 18:53:55.803805 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.803698 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="03faef26-3245-4083-9546-6d2b4657358c" containerName="console" Jul 07 18:53:55.805610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.805594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.808206 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.808184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4ksf5\"" Jul 07 18:53:55.809013 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.808992 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 18:53:55.809013 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.809006 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 18:53:55.817019 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.816986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mfwq"] Jul 07 18:53:55.878675 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.878602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fff\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-kube-api-access-g4fff\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.878859 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.878693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.979419 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.979379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.979590 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.979467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fff\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-kube-api-access-g4fff\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.988831 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.988804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:55.988958 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:55.988860 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fff\" (UniqueName: \"kubernetes.io/projected/6a59b9c2-8e6e-40db-b88a-9e4bcde9c046-kube-api-access-g4fff\") pod \"cert-manager-webhook-6486ffd997-5mfwq\" (UID: \"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:56.127005 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:56.126914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:56.256739 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:56.256703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mfwq"] Jul 07 18:53:56.260167 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:53:56.260123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a59b9c2_8e6e_40db_b88a_9e4bcde9c046.slice/crio-c76173a16d53d756bf5be4eb15c4463b8b5902d403c95925afff8ecc82e21302 WatchSource:0}: Error finding container c76173a16d53d756bf5be4eb15c4463b8b5902d403c95925afff8ecc82e21302: Status 404 returned error can't find the container with id c76173a16d53d756bf5be4eb15c4463b8b5902d403c95925afff8ecc82e21302 Jul 07 18:53:56.669296 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:56.669257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" event={"ID":"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046","Type":"ContainerStarted","Data":"c76173a16d53d756bf5be4eb15c4463b8b5902d403c95925afff8ecc82e21302"} Jul 07 18:53:59.683185 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:59.683148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" event={"ID":"6a59b9c2-8e6e-40db-b88a-9e4bcde9c046","Type":"ContainerStarted","Data":"c48e4ae3d612836d481f4c047a032dd3eb28d52bf5b50ea37577ad0819d905f1"} Jul 07 18:53:59.683583 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:59.683282 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:53:59.700891 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:53:59.700839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" podStartSLOduration=1.468483196 podStartE2EDuration="4.700821936s" totalImagesPullingTime="3.23233874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:53:59.699834299 +0000 UTC m=+211.642457085" watchObservedRunningTime="2026-07-07 18:53:59.700821936 +0000 UTC m=+211.643444723" Jul 07 18:54:05.688962 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:05.688933 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5mfwq" Jul 07 18:54:16.122407 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.122370 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xlpgv"] Jul 07 18:54:16.125131 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.125110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.127188 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.127173 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cnrq9\"" Jul 07 18:54:16.134621 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.134597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xlpgv"] Jul 07 18:54:16.268248 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.268206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczcf\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-kube-api-access-fczcf\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.268417 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.268289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-bound-sa-token\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.368900 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.368858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczcf\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-kube-api-access-fczcf\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.369082 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.368948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-bound-sa-token\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.377346 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.377287 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-bound-sa-token\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.377454 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.377351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczcf\" (UniqueName: \"kubernetes.io/projected/ecd69fb9-4aae-4bc8-83ed-1b250865ea86-kube-api-access-fczcf\") pod \"cert-manager-55b4d954d5-xlpgv\" (UID: \"ecd69fb9-4aae-4bc8-83ed-1b250865ea86\") " pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.434339 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.434295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xlpgv" Jul 07 18:54:16.564458 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.564285 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xlpgv"] Jul 07 18:54:16.567694 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:54:16.567618 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecd69fb9_4aae_4bc8_83ed_1b250865ea86.slice/crio-530ffb14499a986ff85208e32b01134d24cdd84e02b51ba48bd420fe35a8ed1d WatchSource:0}: Error finding container 530ffb14499a986ff85208e32b01134d24cdd84e02b51ba48bd420fe35a8ed1d: Status 404 returned error can't find the container with id 530ffb14499a986ff85208e32b01134d24cdd84e02b51ba48bd420fe35a8ed1d Jul 07 18:54:16.751187 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.751153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xlpgv" event={"ID":"ecd69fb9-4aae-4bc8-83ed-1b250865ea86","Type":"ContainerStarted","Data":"4c5d8d3cb4c6fd42070d7d4fc9410bafed47dac1e1a8ebb1a5f9139123ba95b4"} Jul 07 18:54:16.751356 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.751195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xlpgv" event={"ID":"ecd69fb9-4aae-4bc8-83ed-1b250865ea86","Type":"ContainerStarted","Data":"530ffb14499a986ff85208e32b01134d24cdd84e02b51ba48bd420fe35a8ed1d"} Jul 07 18:54:16.775159 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:16.775096 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xlpgv" podStartSLOduration=0.775078602 podStartE2EDuration="775.078602ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:54:16.773333259 +0000 UTC m=+228.715956047" watchObservedRunningTime="2026-07-07 18:54:16.775078602 +0000 UTC m=+228.717701387" Jul 07 18:54:31.823744 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.823685 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv"] Jul 07 18:54:31.828607 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.828570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:31.831811 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.831788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 18:54:31.831924 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.831886 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gw8f5\"" Jul 07 18:54:31.832184 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.832164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 18:54:31.832939 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.832921 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 18:54:31.836793 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.836758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 18:54:31.841298 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.841277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv"] Jul 07 18:54:31.907230 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.907199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:31.907394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.907251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:31.907394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.907292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-tmp\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:31.907394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.907329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-manifests\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:31.907394 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:31.907349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rkk\" (UniqueName: \"kubernetes.io/projected/b3de752d-ba49-4401-a36b-c78af123304c-kube-api-access-l4rkk\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008044 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008044 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-tmp\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008295 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-manifests\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008295 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rkk\" (UniqueName: \"kubernetes.io/projected/b3de752d-ba49-4401-a36b-c78af123304c-kube-api-access-l4rkk\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008295 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.008574 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.008541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-manifests\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.010456 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.010432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3de752d-ba49-4401-a36b-c78af123304c-tmp\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.010600 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.010583 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.010688 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.010668 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3de752d-ba49-4401-a36b-c78af123304c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.018792 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.018769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rkk\" (UniqueName: \"kubernetes.io/projected/b3de752d-ba49-4401-a36b-c78af123304c-kube-api-access-l4rkk\") pod \"opendatahub-operator-controller-manager-64dd56d745-jf7tv\" (UID: \"b3de752d-ba49-4401-a36b-c78af123304c\") " pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.140011 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.139920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:32.279089 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.278970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv"] Jul 07 18:54:32.281416 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:54:32.281386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3de752d_ba49_4401_a36b_c78af123304c.slice/crio-5eb3182e7eff5582696ab99d4727da6b30919b0fe41e1cec25cb33905a830851 WatchSource:0}: Error finding container 5eb3182e7eff5582696ab99d4727da6b30919b0fe41e1cec25cb33905a830851: Status 404 returned error can't find the container with id 5eb3182e7eff5582696ab99d4727da6b30919b0fe41e1cec25cb33905a830851 Jul 07 18:54:32.817297 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:32.817255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" event={"ID":"b3de752d-ba49-4401-a36b-c78af123304c","Type":"ContainerStarted","Data":"5eb3182e7eff5582696ab99d4727da6b30919b0fe41e1cec25cb33905a830851"} Jul 07 18:54:35.833601 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:35.833563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" event={"ID":"b3de752d-ba49-4401-a36b-c78af123304c","Type":"ContainerStarted","Data":"50a8a596dc2b5d28c0d8aa4388e6cb7b263df10a902ec55d58ea87db4493a39f"} Jul 07 18:54:36.838697 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:36.838663 2564 generic.go:356] "Generic (PLEG): container finished" podID="b3de752d-ba49-4401-a36b-c78af123304c" containerID="50a8a596dc2b5d28c0d8aa4388e6cb7b263df10a902ec55d58ea87db4493a39f" exitCode=0 Jul 07 18:54:36.838697 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:36.838681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" event={"ID":"b3de752d-ba49-4401-a36b-c78af123304c","Type":"ContainerDied","Data":"50a8a596dc2b5d28c0d8aa4388e6cb7b263df10a902ec55d58ea87db4493a39f"} Jul 07 18:54:37.844402 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:37.844361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" event={"ID":"b3de752d-ba49-4401-a36b-c78af123304c","Type":"ContainerStarted","Data":"33c164c11dbc04c4783c8a93ad09c62d372550b09004afdee24f5f2bfdb1709c"} Jul 07 18:54:37.844796 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:37.844463 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:37.869498 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:37.869451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" podStartSLOduration=1.9886465580000001 podStartE2EDuration="6.86943343s" totalImagesPullingTime="3.78580726s" totalInitContainerRuntime="1.094979612s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:54:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:54:37.868464782 +0000 UTC m=+249.811087569" watchObservedRunningTime="2026-07-07 18:54:37.86943343 +0000 UTC m=+249.812056218" Jul 07 18:54:48.851104 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:48.851021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64dd56d745-jf7tv" Jul 07 18:54:53.930948 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.930914 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj"] Jul 07 18:54:53.933337 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.933316 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:53.938426 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 18:54:53.938573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 18:54:53.938573 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 18:54:53.938715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938435 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mg5k7\"" Jul 07 18:54:53.938715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938449 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:54:53.938715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.938500 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 18:54:53.945406 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:53.945374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj"] Jul 07 18:54:54.021787 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.021755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.021787 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.021794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d132bd-e691-4d90-835d-f496cafbcc82-manager-config\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.021990 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.021819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-metrics-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.021990 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.021948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5vrq\" (UniqueName: \"kubernetes.io/projected/54d132bd-e691-4d90-835d-f496cafbcc82-kube-api-access-c5vrq\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.122874 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.122829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d132bd-e691-4d90-835d-f496cafbcc82-manager-config\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.122874 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.122880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-metrics-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.123131 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.122963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vrq\" (UniqueName: \"kubernetes.io/projected/54d132bd-e691-4d90-835d-f496cafbcc82-kube-api-access-c5vrq\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.123131 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.123017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.123444 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.123423 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d132bd-e691-4d90-835d-f496cafbcc82-manager-config\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.125610 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.125585 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.125769 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.125743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d132bd-e691-4d90-835d-f496cafbcc82-metrics-cert\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.132955 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.132930 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vrq\" (UniqueName: \"kubernetes.io/projected/54d132bd-e691-4d90-835d-f496cafbcc82-kube-api-access-c5vrq\") pod \"lws-controller-manager-759f8b6db6-r8gkj\" (UID: \"54d132bd-e691-4d90-835d-f496cafbcc82\") " pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.243176 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.243071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:54.388594 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.388560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj"] Jul 07 18:54:54.392229 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:54:54.392201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d132bd_e691_4d90_835d_f496cafbcc82.slice/crio-5bf8b146c7a940f2c973b1b6e098b50eb10072ddf83831cfd12bcf1b3775fe1a WatchSource:0}: Error finding container 5bf8b146c7a940f2c973b1b6e098b50eb10072ddf83831cfd12bcf1b3775fe1a: Status 404 returned error can't find the container with id 5bf8b146c7a940f2c973b1b6e098b50eb10072ddf83831cfd12bcf1b3775fe1a Jul 07 18:54:54.927686 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:54.927604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" event={"ID":"54d132bd-e691-4d90-835d-f496cafbcc82","Type":"ContainerStarted","Data":"5bf8b146c7a940f2c973b1b6e098b50eb10072ddf83831cfd12bcf1b3775fe1a"} Jul 07 18:54:56.938574 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:56.938525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" event={"ID":"54d132bd-e691-4d90-835d-f496cafbcc82","Type":"ContainerStarted","Data":"9081ca6fdeb66e2b5f9d8095807492722a973b679576ec716174b455afca1252"} Jul 07 18:54:56.939035 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:56.938614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:54:56.957385 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:56.957321 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" podStartSLOduration=1.9505687250000001 podStartE2EDuration="3.957287447s" totalImagesPullingTime="2.006718722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:54:56.956411892 +0000 UTC m=+268.899034678" watchObservedRunningTime="2026-07-07 18:54:56.957287447 +0000 UTC m=+268.899910235" Jul 07 18:54:59.097166 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.097127 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4"] Jul 07 18:54:59.099428 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.099409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.104421 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.104388 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 18:54:59.104561 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.104545 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-948nn\"" Jul 07 18:54:59.108794 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.108773 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 18:54:59.109075 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.109060 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 18:54:59.132437 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.132404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4"] Jul 07 18:54:59.269726 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df10e26-a133-4782-832e-41d2b14a1edd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269877 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269877 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269877 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwr88\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-kube-api-access-pwr88\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.269998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.269947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.270087 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.270000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.270087 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.270018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df10e26-a133-4782-832e-41d2b14a1edd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371305 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371305 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371520 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371520 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371520 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr88\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-kube-api-access-pwr88\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371520 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371754 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371754 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df10e26-a133-4782-832e-41d2b14a1edd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371754 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df10e26-a133-4782-832e-41d2b14a1edd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371754 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371687 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.371951 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371784 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.372011 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.371952 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.372057 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.372005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.372601 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.372577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df10e26-a133-4782-832e-41d2b14a1edd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.373870 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.373850 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df10e26-a133-4782-832e-41d2b14a1edd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.374135 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.374120 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df10e26-a133-4782-832e-41d2b14a1edd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.381269 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.381238 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.382715 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.382695 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwr88\" (UniqueName: \"kubernetes.io/projected/5df10e26-a133-4782-832e-41d2b14a1edd-kube-api-access-pwr88\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jjww4\" (UID: \"5df10e26-a133-4782-832e-41d2b14a1edd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.409587 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.409549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:54:59.584514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.584474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4"] Jul 07 18:54:59.588011 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:54:59.587982 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df10e26_a133_4782_832e_41d2b14a1edd.slice/crio-35b921a32805289221303af5551cb09f50ae08da1db94b8991bab46fe4d50598 WatchSource:0}: Error finding container 35b921a32805289221303af5551cb09f50ae08da1db94b8991bab46fe4d50598: Status 404 returned error can't find the container with id 35b921a32805289221303af5551cb09f50ae08da1db94b8991bab46fe4d50598 Jul 07 18:54:59.958432 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:54:59.958390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" event={"ID":"5df10e26-a133-4782-832e-41d2b14a1edd","Type":"ContainerStarted","Data":"35b921a32805289221303af5551cb09f50ae08da1db94b8991bab46fe4d50598"} Jul 07 18:55:01.812613 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.812575 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb"] Jul 07 18:55:01.818033 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.818009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.821355 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.821326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 18:55:01.821514 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.821376 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h6kqg\"" Jul 07 18:55:01.822151 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.822072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 18:55:01.822151 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.822110 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 18:55:01.843323 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.843294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb"] Jul 07 18:55:01.895953 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.895915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.895994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.896037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.896064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9pn\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-kube-api-access-fc9pn\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896153 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.896116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963bc9a-7bf0-40be-a11c-acda753d9945-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.896152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.896330 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.896216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996772 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9pn\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-kube-api-access-fc9pn\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963bc9a-7bf0-40be-a11c-acda753d9945-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.996964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.997210 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.996993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.997883 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.997854 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.999665 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.999622 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.999826 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.999803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963bc9a-7bf0-40be-a11c-acda753d9945-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:01.999938 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:01.999921 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:02.000029 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.000010 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:02.010023 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.009990 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:02.010169 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.010032 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9pn\" (UniqueName: \"kubernetes.io/projected/1963bc9a-7bf0-40be-a11c-acda753d9945-kube-api-access-fc9pn\") pod \"istiod-openshift-gateway-94bb8fbfd-qlhpb\" (UID: \"1963bc9a-7bf0-40be-a11c-acda753d9945\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:02.106778 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.106729 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:55:02.106885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.106825 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:55:02.106885 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.106859 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:55:02.129716 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.129688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:02.293088 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.293020 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb"] Jul 07 18:55:02.295997 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:55:02.295965 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963bc9a_7bf0_40be_a11c_acda753d9945.slice/crio-ee7e614f93756bbdf697721ae69fa594ca132e5f5d8a498ca054d984e164db52 WatchSource:0}: Error finding container ee7e614f93756bbdf697721ae69fa594ca132e5f5d8a498ca054d984e164db52: Status 404 returned error can't find the container with id ee7e614f93756bbdf697721ae69fa594ca132e5f5d8a498ca054d984e164db52 Jul 07 18:55:02.976729 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.976685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" event={"ID":"1963bc9a-7bf0-40be-a11c-acda753d9945","Type":"ContainerStarted","Data":"ee7e614f93756bbdf697721ae69fa594ca132e5f5d8a498ca054d984e164db52"} Jul 07 18:55:02.978455 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:02.978423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" event={"ID":"5df10e26-a133-4782-832e-41d2b14a1edd","Type":"ContainerStarted","Data":"129345f17b3e0375b132d55e0d7f497f551dc7f5be7d96662a1f78baec7cd61d"} Jul 07 18:55:03.011119 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:03.011070 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" podStartSLOduration=1.494740117 podStartE2EDuration="4.011050598s" totalImagesPullingTime="2.516310481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:55:03.01002971 +0000 UTC m=+274.952652498" watchObservedRunningTime="2026-07-07 18:55:03.011050598 +0000 UTC m=+274.953673386" Jul 07 18:55:03.410478 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:03.410427 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:55:03.416149 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:03.416116 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:55:03.983727 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:03.983682 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:55:03.984957 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:03.984932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jjww4" Jul 07 18:55:04.592597 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:04.592566 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:55:04.592691 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:04.592629 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:55:04.990926 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:04.990814 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" event={"ID":"1963bc9a-7bf0-40be-a11c-acda753d9945","Type":"ContainerStarted","Data":"94f3d5eb1a070e17cfd48592f0c872b09b00276427bf151023101ffe82225a77"} Jul 07 18:55:04.991333 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:04.991000 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:05.077792 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:05.077697 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" podStartSLOduration=1.783525574 podStartE2EDuration="4.077674681s" totalImagesPullingTime="2.294149107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:55:05.07549445 +0000 UTC m=+277.018117238" watchObservedRunningTime="2026-07-07 18:55:05.077674681 +0000 UTC m=+277.020297467" Jul 07 18:55:05.997413 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:05.997379 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qlhpb" Jul 07 18:55:07.945636 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:07.945603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-759f8b6db6-r8gkj" Jul 07 18:55:28.469674 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:55:28.469627 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 18:57:26.867670 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.867611 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j"] Jul 07 18:57:26.870916 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.870891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.873685 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.873631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lhk9t\"" Jul 07 18:57:26.885247 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.885215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j"] Jul 07 18:57:26.900638 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900638 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e56f9a-26a9-4110-be3e-e277710ef5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.900888 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:26.901096 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:26.900887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdvj\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-kube-api-access-wjdvj\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001327 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdvj\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-kube-api-access-wjdvj\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001523 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001828 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001879 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e56f9a-26a9-4110-be3e-e277710ef5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001879 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001863 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.001879 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.002030 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001925 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.002030 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.001975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.002109 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.002067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.002751 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.002727 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e56f9a-26a9-4110-be3e-e277710ef5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.003930 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.003911 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.004194 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.004173 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.009918 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.009894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.010155 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.010134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdvj\" (UniqueName: \"kubernetes.io/projected/09e56f9a-26a9-4110-be3e-e277710ef5dc-kube-api-access-wjdvj\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7q7j\" (UID: \"09e56f9a-26a9-4110-be3e-e277710ef5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.188545 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.188436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:27.329782 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.329590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j"] Jul 07 18:57:27.332273 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:57:27.332242 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e56f9a_26a9_4110_be3e_e277710ef5dc.slice/crio-0bdf053e6758e5719648293245ecfbff765a9dac4080df3beb7febf2ac244d35 WatchSource:0}: Error finding container 0bdf053e6758e5719648293245ecfbff765a9dac4080df3beb7febf2ac244d35: Status 404 returned error can't find the container with id 0bdf053e6758e5719648293245ecfbff765a9dac4080df3beb7febf2ac244d35 Jul 07 18:57:27.334549 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.334508 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:57:27.334637 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.334608 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:57:27.334701 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.334665 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:57:27.641705 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.641663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" event={"ID":"09e56f9a-26a9-4110-be3e-e277710ef5dc","Type":"ContainerStarted","Data":"60fa87f101f454e4a3a0fcd07b349a96c0af339f59a42d1d7f02789ebbfe4223"} Jul 07 18:57:27.641876 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.641731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" event={"ID":"09e56f9a-26a9-4110-be3e-e277710ef5dc","Type":"ContainerStarted","Data":"0bdf053e6758e5719648293245ecfbff765a9dac4080df3beb7febf2ac244d35"} Jul 07 18:57:27.665804 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:27.665743 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" podStartSLOduration=1.6657214009999999 podStartE2EDuration="1.665721401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:57:27.664523343 +0000 UTC m=+419.607146156" watchObservedRunningTime="2026-07-07 18:57:27.665721401 +0000 UTC m=+419.608344190" Jul 07 18:57:28.188909 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:28.188866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:28.193705 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:28.193678 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:28.646544 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:28.646511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:28.647475 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:28.647461 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:57:29.201798 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.201757 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:57:29.204460 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.204436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.206880 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.206857 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-862l7\"" Jul 07 18:57:29.206880 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.206873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 18:57:29.206880 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.206860 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 18:57:29.207122 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.206907 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 18:57:29.215935 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.215903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:57:29.217304 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.217274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f50c866-4337-4b6f-8bb3-53f7bcc18446-config-file\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.217429 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.217324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4c8x\" (UniqueName: \"kubernetes.io/projected/8f50c866-4337-4b6f-8bb3-53f7bcc18446-kube-api-access-k4c8x\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.236181 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.232310 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:57:29.318785 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.318739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f50c866-4337-4b6f-8bb3-53f7bcc18446-config-file\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.318998 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.318809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4c8x\" (UniqueName: \"kubernetes.io/projected/8f50c866-4337-4b6f-8bb3-53f7bcc18446-kube-api-access-k4c8x\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.319375 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.319351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f50c866-4337-4b6f-8bb3-53f7bcc18446-config-file\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.328543 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.328506 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4c8x\" (UniqueName: \"kubernetes.io/projected/8f50c866-4337-4b6f-8bb3-53f7bcc18446-kube-api-access-k4c8x\") pod \"limitador-limitador-54d6d988d5-7gmkd\" (UID: \"8f50c866-4337-4b6f-8bb3-53f7bcc18446\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.517498 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.517401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:29.670177 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.669923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:57:29.672704 ip-10-0-132-43 kubenswrapper[2564]: W0707 18:57:29.672673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f50c866_4337_4b6f_8bb3_53f7bcc18446.slice/crio-610357e06fe4a2414910f8c825c6b0c9b388a9c56700165df34b6000184b34d7 WatchSource:0}: Error finding container 610357e06fe4a2414910f8c825c6b0c9b388a9c56700165df34b6000184b34d7: Status 404 returned error can't find the container with id 610357e06fe4a2414910f8c825c6b0c9b388a9c56700165df34b6000184b34d7 Jul 07 18:57:29.675154 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:29.675134 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:57:30.658570 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:30.658529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" event={"ID":"8f50c866-4337-4b6f-8bb3-53f7bcc18446","Type":"ContainerStarted","Data":"610357e06fe4a2414910f8c825c6b0c9b388a9c56700165df34b6000184b34d7"} Jul 07 18:57:32.669858 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:32.669816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" event={"ID":"8f50c866-4337-4b6f-8bb3-53f7bcc18446","Type":"ContainerStarted","Data":"a1b037e17c3c34fdb22947424f62272229e0d1decbe66fa557e99b3e004e61d4"} Jul 07 18:57:32.675612 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:32.670511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:57:32.691645 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:32.691588 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" podStartSLOduration=0.898688148 podStartE2EDuration="3.69156999s" totalImagesPullingTime="2.792881842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:57:32.689993535 +0000 UTC m=+424.632616326" watchObservedRunningTime="2026-07-07 18:57:32.69156999 +0000 UTC m=+424.634192778" Jul 07 18:57:43.676007 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:57:43.675974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7gmkd" Jul 07 18:58:45.306943 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:45.306901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:58:53.057455 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.057417 2564 generic.go:356] "Generic (PLEG): container finished" podID="09e56f9a-26a9-4110-be3e-e277710ef5dc" containerID="60fa87f101f454e4a3a0fcd07b349a96c0af339f59a42d1d7f02789ebbfe4223" exitCode=0 Jul 07 18:58:53.057901 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.057489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" event={"ID":"09e56f9a-26a9-4110-be3e-e277710ef5dc","Type":"ContainerDied","Data":"60fa87f101f454e4a3a0fcd07b349a96c0af339f59a42d1d7f02789ebbfe4223"} Jul 07 18:58:53.057970 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.057954 2564 scope.go:117] "RemoveContainer" containerID="60fa87f101f454e4a3a0fcd07b349a96c0af339f59a42d1d7f02789ebbfe4223" Jul 07 18:58:53.058964 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.058932 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:58:53.059082 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.059014 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:58:53.059082 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:53.059065 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:58:54.064483 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:54.064453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" event={"ID":"09e56f9a-26a9-4110-be3e-e277710ef5dc","Type":"ContainerStarted","Data":"4eceea0c2cc0d9b36becb70e57abc9620ed621388278f1ce1f087ba65e297ac0"} Jul 07 18:58:54.189140 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:54.189101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:58:54.190124 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:54.190092 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 18:58:54.190246 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:54.190161 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" podUID="09e56f9a-26a9-4110-be3e-e277710ef5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 18:58:55.189360 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:55.189323 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 18:58:55.189800 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:55.189385 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" podUID="09e56f9a-26a9-4110-be3e-e277710ef5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 18:58:56.189799 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:56.189757 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 18:58:56.190192 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:56.189818 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" podUID="09e56f9a-26a9-4110-be3e-e277710ef5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 18:58:57.189958 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:57.189916 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 18:58:57.190351 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:57.189975 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" podUID="09e56f9a-26a9-4110-be3e-e277710ef5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 18:58:58.193553 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:58.193515 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:58:58.193989 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:58.193780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:58:58.194491 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:58:58.194472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7q7j" Jul 07 18:59:10.940470 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:10.940432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:59:22.318550 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:22.318512 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:59:33.222787 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:33.222746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:59:41.314799 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:41.314761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:59:47.017107 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:47.017070 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 18:59:55.218987 ip-10-0-132-43 kubenswrapper[2564]: I0707 18:59:55.218947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:00:00.136914 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.136882 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724180-spr8m"] Jul 07 19:00:00.139393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.139375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:00.141836 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.141805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:00:00.148118 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.148088 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724180-spr8m"] Jul 07 19:00:00.252067 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.252033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq6t\" (UniqueName: \"kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t\") pod \"maas-api-key-cleanup-29724180-spr8m\" (UID: \"f4079b47-851a-403d-8356-4485e73db34d\") " pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:00.352919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.352877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq6t\" (UniqueName: \"kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t\") pod \"maas-api-key-cleanup-29724180-spr8m\" (UID: \"f4079b47-851a-403d-8356-4485e73db34d\") " pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:00.363124 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.363084 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq6t\" (UniqueName: \"kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t\") pod \"maas-api-key-cleanup-29724180-spr8m\" (UID: \"f4079b47-851a-403d-8356-4485e73db34d\") " pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:00.450699 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.450567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:00.586695 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:00.586669 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724180-spr8m"] Jul 07 19:00:00.589149 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:00:00.589116 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4079b47_851a_403d_8356_4485e73db34d.slice/crio-07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e WatchSource:0}: Error finding container 07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e: Status 404 returned error can't find the container with id 07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e Jul 07 19:00:01.377862 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:01.377829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" event={"ID":"f4079b47-851a-403d-8356-4485e73db34d","Type":"ContainerStarted","Data":"07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e"} Jul 07 19:00:06.402097 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:06.402060 2564 generic.go:356] "Generic (PLEG): container finished" podID="f4079b47-851a-403d-8356-4485e73db34d" containerID="b2e7f834910efd799e02a8f11694b56afcb1be81a38a31a7866e915e9f0616d6" exitCode=0 Jul 07 19:00:06.402097 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:06.402102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" event={"ID":"f4079b47-851a-403d-8356-4485e73db34d","Type":"ContainerDied","Data":"b2e7f834910efd799e02a8f11694b56afcb1be81a38a31a7866e915e9f0616d6"} Jul 07 19:00:07.541034 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:07.541005 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:07.725288 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:07.725190 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khq6t\" (UniqueName: \"kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t\") pod \"f4079b47-851a-403d-8356-4485e73db34d\" (UID: \"f4079b47-851a-403d-8356-4485e73db34d\") " Jul 07 19:00:07.727500 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:07.727469 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t" (OuterVolumeSpecName: "kube-api-access-khq6t") pod "f4079b47-851a-403d-8356-4485e73db34d" (UID: "f4079b47-851a-403d-8356-4485e73db34d"). InnerVolumeSpecName "kube-api-access-khq6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:00:07.826078 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:07.826041 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq6t\" (UniqueName: \"kubernetes.io/projected/f4079b47-851a-403d-8356-4485e73db34d-kube-api-access-khq6t\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:00:08.412617 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:08.412588 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" Jul 07 19:00:08.412812 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:08.412618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724180-spr8m" event={"ID":"f4079b47-851a-403d-8356-4485e73db34d","Type":"ContainerDied","Data":"07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e"} Jul 07 19:00:08.412812 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:00:08.412686 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c498e3edd4f1abefcbb59412261e8747c36bada9404ec3f7365c5da1ec3c7e" Jul 07 19:01:34.336907 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:01:34.336872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:01:44.719737 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:01:44.719696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:01:55.924008 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:01:55.923973 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:02:03.510801 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:02:03.510758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:02:11.912143 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:02:11.912109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:02:19.334344 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:02:19.334311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:02:42.124875 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:02:42.124836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:02:53.340670 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:02:53.340604 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:03:29.214796 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:03:29.214758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:03:37.938640 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:03:37.938597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:03:47.728520 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:03:47.728477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:03:55.614736 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:03:55.614698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:04:05.918904 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:04:05.918867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:04:17.022826 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:04:17.022784 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:04:27.714542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:04:27.714503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:04:37.318722 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:04:37.318676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:05:00.112919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:05:00.112879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:05:11.828122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:05:11.828079 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:05:35.319406 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:05:35.319368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:05:46.923752 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:05:46.923709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:06:01.928919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:06:01.928881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:06:12.822953 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:06:12.822916 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:06:21.262910 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:06:21.262872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:06:30.160432 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:06:30.160394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:06:39.717731 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:06:39.717688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:08:28.852056 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:08:28.852018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:08:38.549155 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:08:38.549112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:08:50.945701 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:08:50.945659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:00.236533 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:00.236493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:11.367836 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:11.367794 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:22.534966 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:22.534925 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:31.933137 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:31.933101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:42.428372 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:42.428335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:09:52.930254 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:09:52.930164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:00.830150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:00.830114 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:09.837060 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:09.837021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:18.825763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:18.825725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:37.241463 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:37.241417 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:46.722921 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:46.722882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:10:57.930428 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:10:57.930385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:11:09.522546 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:11:09.522494 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:11:24.825768 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:11:24.825668 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:11:33.726901 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:11:33.726860 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:11:43.016048 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:11:43.016011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:11:52.728730 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:11:52.728690 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:01.018990 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:01.018952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:09.815537 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:09.815496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:23.211791 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:23.211750 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:32.422324 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:32.422284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:43.633885 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:43.633847 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:12:54.434933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:12:54.434841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:13:07.045531 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:13:07.045494 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:13:15.623147 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:13:15.623109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:13:28.022343 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:13:28.022301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:13:38.720518 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:13:38.720474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:13:51.314470 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:13:51.314432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:00.217317 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:00.217281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:13.016400 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:13.016352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:21.316020 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:21.315935 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:27.576970 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.576930 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:27.577354 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.577342 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4079b47-851a-403d-8356-4485e73db34d" containerName="cleanup" Jul 07 19:14:27.577402 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.577357 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4079b47-851a-403d-8356-4485e73db34d" containerName="cleanup" Jul 07 19:14:27.577435 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.577425 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4079b47-851a-403d-8356-4485e73db34d" containerName="cleanup" Jul 07 19:14:27.579611 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.579593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.582419 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.582393 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f11be0a0-openshift-default-dockercfg-t77nt\"" Jul 07 19:14:27.656018 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.655979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:27.689234 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689580 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689580 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.689580 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.689544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb22\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790558 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790558 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.790829 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb22\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.790853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.791051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.791092 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791270 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.791235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791409 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.791390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.791522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.791473 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.793310 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.793284 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.793455 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.793390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.801808 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.801771 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.802046 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.802024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb22\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22\") pod \"e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:27.890394 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:27.890294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:28.046424 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.046400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:28.049298 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:14:28.049268 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ab27bb_495e_4c5d_9c08_f44d41680acf.slice/crio-8cb24bda6066cd4c359d11f356dbbd7d2f88562ef2d166e6121fa46753229f6c WatchSource:0}: Error finding container 8cb24bda6066cd4c359d11f356dbbd7d2f88562ef2d166e6121fa46753229f6c: Status 404 returned error can't find the container with id 8cb24bda6066cd4c359d11f356dbbd7d2f88562ef2d166e6121fa46753229f6c Jul 07 19:14:28.051416 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.051377 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:28.051504 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.051456 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:28.051504 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.051483 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:28.503062 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.503029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" event={"ID":"30ab27bb-495e-4c5d-9c08-f44d41680acf","Type":"ContainerStarted","Data":"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b"} Jul 07 19:14:28.503239 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.503080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" event={"ID":"30ab27bb-495e-4c5d-9c08-f44d41680acf","Type":"ContainerStarted","Data":"8cb24bda6066cd4c359d11f356dbbd7d2f88562ef2d166e6121fa46753229f6c"} Jul 07 19:14:28.524697 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.524619 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" podStartSLOduration=1.524599029 podStartE2EDuration="1.524599029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:28.523537485 +0000 UTC m=+1440.466160273" watchObservedRunningTime="2026-07-07 19:14:28.524599029 +0000 UTC m=+1440.467221816" Jul 07 19:14:28.859140 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.859093 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:28.861705 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.861686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:28.864157 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.864123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f11be0a0-serving-cert\"" Jul 07 19:14:28.864291 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.864252 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:14:28.874996 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.874949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:28.890386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.890357 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:28.899812 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.899766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:28.900004 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.899819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8km\" (UniqueName: \"kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:28.900004 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:28.899978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.001364 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.001316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.001575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.001421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.001575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.001456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8km\" (UniqueName: \"kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.001575 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:29.001475 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f11be0a0-serving-cert: secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:29.001575 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:29.001546 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls podName:c9c6cd0a-c832-4317-831a-5fcc75f2e1b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:14:29.501523166 +0000 UTC m=+1441.444145947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls") pod "maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0") : secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:29.002189 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.002161 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.018894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.018859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8km\" (UniqueName: \"kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.506764 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.506727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.510089 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.510054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") pod \"maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.773132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.773022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:29.896139 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.896106 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:29.913600 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.913577 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:29.915635 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:14:29.915608 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c6cd0a_c832_4317_831a_5fcc75f2e1b0.slice/crio-8cb21ed3653750ac8786e77a1f67b8777a3960ff8b6a5e0e63c8ae2692d99bc8 WatchSource:0}: Error finding container 8cb21ed3653750ac8786e77a1f67b8777a3960ff8b6a5e0e63c8ae2692d99bc8: Status 404 returned error can't find the container with id 8cb21ed3653750ac8786e77a1f67b8777a3960ff8b6a5e0e63c8ae2692d99bc8 Jul 07 19:14:29.917102 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:29.917057 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:14:30.515853 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:30.515818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" event={"ID":"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0","Type":"ContainerStarted","Data":"8cb21ed3653750ac8786e77a1f67b8777a3960ff8b6a5e0e63c8ae2692d99bc8"} Jul 07 19:14:30.516082 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:30.515988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:30.516933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:30.516915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:31.029065 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:31.029019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:32.531967 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:32.531927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" event={"ID":"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0","Type":"ContainerStarted","Data":"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32"} Jul 07 19:14:32.532789 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:32.532770 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:32.550444 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:32.550384 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" podStartSLOduration=2.398270069 podStartE2EDuration="4.550366437s" totalImagesPullingTime="2.152096368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:32.54979808 +0000 UTC m=+1444.492420885" watchObservedRunningTime="2026-07-07 19:14:32.550366437 +0000 UTC m=+1444.492989225" Jul 07 19:14:34.054469 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:34.054432 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:34.055020 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:34.054672 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" podUID="30ab27bb-495e-4c5d-9c08-f44d41680acf" containerName="istio-proxy" containerID="cri-o://8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b" gracePeriod=30 Jul 07 19:14:34.345385 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:34.345254 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:34.668505 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:34.668410 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f11be0a0-serving-cert: secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:34.668505 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:34.668488 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls podName:c9c6cd0a-c832-4317-831a-5fcc75f2e1b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:14:35.168470651 +0000 UTC m=+1447.111093417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls") pod "maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0") : secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:35.174069 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:35.174025 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f11be0a0-serving-cert: secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:35.174585 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:35.174119 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls podName:c9c6cd0a-c832-4317-831a-5fcc75f2e1b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:14:36.17409684 +0000 UTC m=+1448.116719606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls") pod "maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0") : secret "maas-api-e2e-ait-f11be0a0-serving-cert" not found Jul 07 19:14:35.547542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.547498 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" podUID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" containerName="maas-api" containerID="cri-o://fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32" gracePeriod=30 Jul 07 19:14:35.845981 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.845950 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:35.983362 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.983314 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8km\" (UniqueName: \"kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km\") pod \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " Jul 07 19:14:35.983575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.983405 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca\") pod \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " Jul 07 19:14:35.983575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.983514 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") pod \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\" (UID: \"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0\") " Jul 07 19:14:35.983745 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.983723 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:14:35.983894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.983878 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:35.985625 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.985602 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:14:35.985715 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:35.985635 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km" (OuterVolumeSpecName: "kube-api-access-5t8km") pod "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" (UID: "c9c6cd0a-c832-4317-831a-5fcc75f2e1b0"). InnerVolumeSpecName "kube-api-access-5t8km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:36.084891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.084809 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:36.084891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.084839 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8km\" (UniqueName: \"kubernetes.io/projected/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0-kube-api-access-5t8km\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:36.553922 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.553875 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" containerID="fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32" exitCode=0 Jul 07 19:14:36.553922 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.553923 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" Jul 07 19:14:36.554464 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.553955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" event={"ID":"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0","Type":"ContainerDied","Data":"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32"} Jul 07 19:14:36.554464 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.554023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6" event={"ID":"c9c6cd0a-c832-4317-831a-5fcc75f2e1b0","Type":"ContainerDied","Data":"8cb21ed3653750ac8786e77a1f67b8777a3960ff8b6a5e0e63c8ae2692d99bc8"} Jul 07 19:14:36.554464 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.554046 2564 scope.go:117] "RemoveContainer" containerID="fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32" Jul 07 19:14:36.564511 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.564488 2564 scope.go:117] "RemoveContainer" containerID="fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32" Jul 07 19:14:36.564912 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:36.564892 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32\": container with ID starting with fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32 not found: ID does not exist" containerID="fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32" Jul 07 19:14:36.565014 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.564924 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32"} err="failed to get container status \"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32\": rpc error: code = NotFound desc = could not find container \"fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32\": container with ID starting with fd070e586dbee5d869358928f2c75bd3f3841061f4de93dfb2db595bee4c4c32 not found: ID does not exist" Jul 07 19:14:36.579879 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.579838 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:36.584208 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.584171 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f11be0a0-5c9fb45b55-8hch6"] Jul 07 19:14:36.621280 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:36.621242 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" path="/var/lib/kubelet/pods/c9c6cd0a-c832-4317-831a-5fcc75f2e1b0/volumes" Jul 07 19:14:38.915728 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:38.915701 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:39.014367 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014266 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wb22\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014367 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014313 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014367 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014692 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014692 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014403 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014692 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014692 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014555 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.014883 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.014796 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.015769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.015413 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:39.015769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.015457 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:14:39.015769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.015469 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:39.015769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.015674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket\") pod \"30ab27bb-495e-4c5d-9c08-f44d41680acf\" (UID: \"30ab27bb-495e-4c5d-9c08-f44d41680acf\") " Jul 07 19:14:39.016079 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016023 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:39.016199 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016084 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data" (OuterVolumeSpecName: "istio-data") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:39.017202 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016901 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ab27bb-495e-4c5d-9c08-f44d41680acf-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.017202 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016951 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.017202 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016967 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.017202 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.016986 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.017202 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.017001 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.017582 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.017267 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22" (OuterVolumeSpecName: "kube-api-access-9wb22") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "kube-api-access-9wb22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:39.018085 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.018040 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token" (OuterVolumeSpecName: "istio-token") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:39.023912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.023881 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:39.023912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.023885 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30ab27bb-495e-4c5d-9c08-f44d41680acf" (UID: "30ab27bb-495e-4c5d-9c08-f44d41680acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:14:39.118355 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.118310 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.118355 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.118345 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wb22\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-kube-api-access-9wb22\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.118355 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.118356 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.118355 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.118366 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ab27bb-495e-4c5d-9c08-f44d41680acf-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:39.572540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.572502 2564 generic.go:356] "Generic (PLEG): container finished" podID="30ab27bb-495e-4c5d-9c08-f44d41680acf" containerID="8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b" exitCode=0 Jul 07 19:14:39.572771 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.572575 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" Jul 07 19:14:39.572771 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.572598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" event={"ID":"30ab27bb-495e-4c5d-9c08-f44d41680acf","Type":"ContainerDied","Data":"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b"} Jul 07 19:14:39.572771 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.572742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb" event={"ID":"30ab27bb-495e-4c5d-9c08-f44d41680acf","Type":"ContainerDied","Data":"8cb24bda6066cd4c359d11f356dbbd7d2f88562ef2d166e6121fa46753229f6c"} Jul 07 19:14:39.572948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.572775 2564 scope.go:117] "RemoveContainer" containerID="8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b" Jul 07 19:14:39.582161 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.582141 2564 scope.go:117] "RemoveContainer" containerID="8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b" Jul 07 19:14:39.582456 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:39.582435 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b\": container with ID starting with 8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b not found: ID does not exist" containerID="8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b" Jul 07 19:14:39.582530 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.582465 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b"} err="failed to get container status \"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b\": rpc error: code = NotFound desc = could not find container \"8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b\": container with ID starting with 8d7aafc85bb25fc72580a2abcff9186167ee054988c301b3f42fe6289003890b not found: ID does not exist" Jul 07 19:14:39.594848 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.594811 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:39.599885 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.599852 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f11be0a0-openshift-default-59fb6fd47f-p8jcb"] Jul 07 19:14:39.826095 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:39.825993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:14:40.620571 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.620536 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ab27bb-495e-4c5d-9c08-f44d41680acf" path="/var/lib/kubelet/pods/30ab27bb-495e-4c5d-9c08-f44d41680acf/volumes" Jul 07 19:14:40.695442 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.695403 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:40.696211 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696181 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" containerName="maas-api" Jul 07 19:14:40.696211 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696214 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" containerName="maas-api" Jul 07 19:14:40.696404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696232 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ab27bb-495e-4c5d-9c08-f44d41680acf" containerName="istio-proxy" Jul 07 19:14:40.696404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696243 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ab27bb-495e-4c5d-9c08-f44d41680acf" containerName="istio-proxy" Jul 07 19:14:40.696404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696319 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ab27bb-495e-4c5d-9c08-f44d41680acf" containerName="istio-proxy" Jul 07 19:14:40.696404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.696330 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c6cd0a-c832-4317-831a-5fcc75f2e1b0" containerName="maas-api" Jul 07 19:14:40.699366 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.699344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.701864 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.701839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ed2afad7-openshift-default-dockercfg-z7m8g\"" Jul 07 19:14:40.712745 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.712698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:40.836759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xxv\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.836984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.836970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.837209 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.837060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.937894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.937894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.937894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.937894 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xxv\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.937980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938439 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938272 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938439 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938439 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938581 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938567 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.938713 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.938693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.940463 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.940438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.940782 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.940761 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.947953 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.947927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:40.948117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:40.948094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xxv\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv\") pod \"e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:41.012057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.012016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:41.159985 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.159838 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:41.163022 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:14:41.162991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fd4b3d_124e_47d3_abe8_7412264e4f3e.slice/crio-63f296b4c2607253b8d35be89ab97f6fa4d96b4c4f20e7ad694aff7220ee34f1 WatchSource:0}: Error finding container 63f296b4c2607253b8d35be89ab97f6fa4d96b4c4f20e7ad694aff7220ee34f1: Status 404 returned error can't find the container with id 63f296b4c2607253b8d35be89ab97f6fa4d96b4c4f20e7ad694aff7220ee34f1 Jul 07 19:14:41.165633 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.165597 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:41.165740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.165693 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:41.165740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.165727 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:41.586161 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.586123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerStarted","Data":"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87"} Jul 07 19:14:41.586368 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.586185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerStarted","Data":"63f296b4c2607253b8d35be89ab97f6fa4d96b4c4f20e7ad694aff7220ee34f1"} Jul 07 19:14:41.610337 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:41.610286 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" podStartSLOduration=1.610267741 podStartE2EDuration="1.610267741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:41.609671312 +0000 UTC m=+1453.552294101" watchObservedRunningTime="2026-07-07 19:14:41.610267741 +0000 UTC m=+1453.552890520" Jul 07 19:14:42.012832 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.012725 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:42.018440 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.018407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:42.033247 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.033212 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:42.036608 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.036581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.039971 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.039944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-ed2afad7-serving-cert\"" Jul 07 19:14:42.040764 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.040737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:14:42.072075 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.072043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:42.151637 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.151585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.151946 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.151721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnmx\" (UniqueName: \"kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.151946 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.151766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.253432 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.253378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.253688 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.253488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnmx\" (UniqueName: \"kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.253688 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.253527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.254077 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.254056 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.256208 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.256181 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.262352 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.262323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnmx\" (UniqueName: \"kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx\") pod \"maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.352290 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.352247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:42.502363 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.502321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:42.503325 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:14:42.503289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898e45c1_4da6_4de9_887a_aa114e0a2cd5.slice/crio-cb478d0bc3b05c244f6dd5d877b49bdc7788ee0639b5b8f06336b37b6567842e WatchSource:0}: Error finding container cb478d0bc3b05c244f6dd5d877b49bdc7788ee0639b5b8f06336b37b6567842e: Status 404 returned error can't find the container with id cb478d0bc3b05c244f6dd5d877b49bdc7788ee0639b5b8f06336b37b6567842e Jul 07 19:14:42.592354 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.592308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" event={"ID":"898e45c1-4da6-4de9-887a-aa114e0a2cd5","Type":"ContainerStarted","Data":"cb478d0bc3b05c244f6dd5d877b49bdc7788ee0639b5b8f06336b37b6567842e"} Jul 07 19:14:42.592562 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.592532 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:42.593564 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:42.593542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:43.599697 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:43.599658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" event={"ID":"898e45c1-4da6-4de9-887a-aa114e0a2cd5","Type":"ContainerStarted","Data":"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44"} Jul 07 19:14:43.624143 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:43.624089 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" podStartSLOduration=1.010840859 podStartE2EDuration="1.624070392s" totalImagesPullingTime="613.229533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:43.621132023 +0000 UTC m=+1455.563754812" watchObservedRunningTime="2026-07-07 19:14:43.624070392 +0000 UTC m=+1455.566693178" Jul 07 19:14:44.605513 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:44.605422 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:47.323687 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.314644 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:47.324713 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.324667 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" podUID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" containerName="maas-api" containerID="cri-o://4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44" gracePeriod=30 Jul 07 19:14:47.598311 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.598283 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:47.648852 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.648809 2564 generic.go:356] "Generic (PLEG): container finished" podID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" containerID="4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44" exitCode=0 Jul 07 19:14:47.649057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.648878 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" Jul 07 19:14:47.649057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.648890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" event={"ID":"898e45c1-4da6-4de9-887a-aa114e0a2cd5","Type":"ContainerDied","Data":"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44"} Jul 07 19:14:47.649057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.648958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz" event={"ID":"898e45c1-4da6-4de9-887a-aa114e0a2cd5","Type":"ContainerDied","Data":"cb478d0bc3b05c244f6dd5d877b49bdc7788ee0639b5b8f06336b37b6567842e"} Jul 07 19:14:47.649057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.648990 2564 scope.go:117] "RemoveContainer" containerID="4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44" Jul 07 19:14:47.658324 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.658300 2564 scope.go:117] "RemoveContainer" containerID="4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44" Jul 07 19:14:47.658697 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:47.658675 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44\": container with ID starting with 4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44 not found: ID does not exist" containerID="4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44" Jul 07 19:14:47.658788 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.658713 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44"} err="failed to get container status \"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44\": rpc error: code = NotFound desc = could not find container \"4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44\": container with ID starting with 4f206119612a0e25b0ffb6f4caf6ba79c80417da9e3ee6ac4bfa84198c0dfa44 not found: ID does not exist" Jul 07 19:14:47.729676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.729013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca\") pod \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " Jul 07 19:14:47.729676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.729100 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls\") pod \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " Jul 07 19:14:47.729676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.729182 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trnmx\" (UniqueName: \"kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx\") pod \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\" (UID: \"898e45c1-4da6-4de9-887a-aa114e0a2cd5\") " Jul 07 19:14:47.730365 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.730185 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "898e45c1-4da6-4de9-887a-aa114e0a2cd5" (UID: "898e45c1-4da6-4de9-887a-aa114e0a2cd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:14:47.733196 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.733147 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "898e45c1-4da6-4de9-887a-aa114e0a2cd5" (UID: "898e45c1-4da6-4de9-887a-aa114e0a2cd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:14:47.742639 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.742601 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx" (OuterVolumeSpecName: "kube-api-access-trnmx") pod "898e45c1-4da6-4de9-887a-aa114e0a2cd5" (UID: "898e45c1-4da6-4de9-887a-aa114e0a2cd5"). InnerVolumeSpecName "kube-api-access-trnmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:47.831063 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.830957 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnmx\" (UniqueName: \"kubernetes.io/projected/898e45c1-4da6-4de9-887a-aa114e0a2cd5-kube-api-access-trnmx\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:47.831296 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.831279 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898e45c1-4da6-4de9-887a-aa114e0a2cd5-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:47.831417 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.831404 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898e45c1-4da6-4de9-887a-aa114e0a2cd5-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:47.974676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.971781 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:47.979544 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:47.979510 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-ed2afad7-557f5dbb65-495kz"] Jul 07 19:14:48.624338 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:48.624287 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" path="/var/lib/kubelet/pods/898e45c1-4da6-4de9-887a-aa114e0a2cd5/volumes" Jul 07 19:14:52.164990 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:52.164954 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fd4b3d_124e_47d3_abe8_7412264e4f3e.slice/crio-425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87.scope\": RecentStats: unable to find data in memory cache]" Jul 07 19:14:52.675247 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.675217 2564 generic.go:356] "Generic (PLEG): container finished" podID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerID="425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87" exitCode=0 Jul 07 19:14:52.675430 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.675294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerDied","Data":"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87"} Jul 07 19:14:52.675755 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.675740 2564 scope.go:117] "RemoveContainer" containerID="425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87" Jul 07 19:14:52.676912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.676883 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:52.676985 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.676971 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:52.677020 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:52.677011 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:53.016615 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:53.016521 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:53.681500 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:53.681463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerStarted","Data":"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5"} Jul 07 19:14:53.681991 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:53.681526 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" containerID="cri-o://c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5" gracePeriod=30 Jul 07 19:14:57.593562 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:57.593513 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:58.970897 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:58.970859 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:59.041973 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.041937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7xxv\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042094 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042150 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042132 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042414 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042178 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042414 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042414 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042358 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs\") pod \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\" (UID: \"35fd4b3d-124e-47d3-abe8-7412264e4f3e\") " Jul 07 19:14:59.042571 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042426 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:59.042571 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042461 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:59.042745 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042722 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:59.042834 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042819 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.042887 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042843 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.042959 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.042921 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:14:59.043090 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.043065 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:59.044471 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.044445 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:14:59.044686 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.044643 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:59.044788 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.044769 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv" (OuterVolumeSpecName: "kube-api-access-b7xxv") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "kube-api-access-b7xxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:59.044788 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.044780 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35fd4b3d-124e-47d3-abe8-7412264e4f3e" (UID: "35fd4b3d-124e-47d3-abe8-7412264e4f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:59.143386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143340 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143376 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143385 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143392 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143400 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143730 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143408 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xxv\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-kube-api-access-b7xxv\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.143730 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.143417 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd4b3d-124e-47d3-abe8-7412264e4f3e-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:14:59.557836 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.557549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:14:59.558125 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558101 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:14:59.558125 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558123 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:14:59.558302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558155 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" containerName="maas-api" Jul 07 19:14:59.558302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558164 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" containerName="maas-api" Jul 07 19:14:59.558302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558230 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:14:59.558302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558240 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="898e45c1-4da6-4de9-887a-aa114e0a2cd5" containerName="maas-api" Jul 07 19:14:59.558302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.558248 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:14:59.561639 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.561619 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.568854 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.565976 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-914b2894-openshift-default-dockercfg-l8g8s\"" Jul 07 19:14:59.573972 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.573936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:14:59.648525 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648525 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc57b\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648916 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648916 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.648916 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.648861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.712452 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.712412 2564 generic.go:356] "Generic (PLEG): container finished" podID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerID="c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5" exitCode=0 Jul 07 19:14:59.712633 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.712468 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" Jul 07 19:14:59.712633 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.712489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerDied","Data":"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5"} Jul 07 19:14:59.712633 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.712549 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq" event={"ID":"35fd4b3d-124e-47d3-abe8-7412264e4f3e","Type":"ContainerDied","Data":"63f296b4c2607253b8d35be89ab97f6fa4d96b4c4f20e7ad694aff7220ee34f1"} Jul 07 19:14:59.712633 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.712579 2564 scope.go:117] "RemoveContainer" containerID="c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5" Jul 07 19:14:59.722490 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.722470 2564 scope.go:117] "RemoveContainer" containerID="425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87" Jul 07 19:14:59.731089 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.731067 2564 scope.go:117] "RemoveContainer" containerID="c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5" Jul 07 19:14:59.731393 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:59.731374 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5\": container with ID starting with c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5 not found: ID does not exist" containerID="c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5" Jul 07 19:14:59.731471 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.731400 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5"} err="failed to get container status \"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5\": rpc error: code = NotFound desc = could not find container \"c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5\": container with ID starting with c21a36e8879d91ced392af2650da6c7765bb13069fb3283517d92cd975cbd9c5 not found: ID does not exist" Jul 07 19:14:59.731471 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.731418 2564 scope.go:117] "RemoveContainer" containerID="425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87" Jul 07 19:14:59.731662 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:14:59.731633 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87\": container with ID starting with 425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87 not found: ID does not exist" containerID="425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87" Jul 07 19:14:59.731707 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.731667 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87"} err="failed to get container status \"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87\": rpc error: code = NotFound desc = could not find container \"425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87\": container with ID starting with 425bbcebfd872c0ee76623a9569c89900829002d239994b76ece46aba3e57e87 not found: ID does not exist" Jul 07 19:14:59.735729 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.735697 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:59.743280 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.743252 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ed2afad7-openshift-default-9f89b84c-bhdzq"] Jul 07 19:14:59.750314 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750401 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750401 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750401 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc57b\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750765 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750823 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.750823 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.750787 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.751090 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.751058 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.751090 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.751072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.751414 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.751232 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.752952 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.752929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.753035 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.752958 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.758794 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.758764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.759105 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.759089 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc57b\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b\") pod \"e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:14:59.877027 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:14:59.876935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:00.024065 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.024015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:15:00.029290 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:15:00.029255 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddb18ac_2704_484d_aca2_fea42d9f141e.slice/crio-d85c4f1ba53d8792a89e53e98a43f173b22b3b33ef6d100a1a3fa58d8a630426 WatchSource:0}: Error finding container d85c4f1ba53d8792a89e53e98a43f173b22b3b33ef6d100a1a3fa58d8a630426: Status 404 returned error can't find the container with id d85c4f1ba53d8792a89e53e98a43f173b22b3b33ef6d100a1a3fa58d8a630426 Jul 07 19:15:00.031624 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.031583 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:15:00.031759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.031691 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:15:00.031759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.031736 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:15:00.133181 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.133084 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-fh82w"] Jul 07 19:15:00.133537 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.133522 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:15:00.133582 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.133540 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" containerName="istio-proxy" Jul 07 19:15:00.136869 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.136849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:00.139766 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.139740 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:15:00.146563 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.146342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-fh82w"] Jul 07 19:15:00.255452 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.255406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7dn\" (UniqueName: \"kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn\") pod \"maas-api-key-cleanup-29724195-fh82w\" (UID: \"c4ccb445-58b0-4f7e-b8fc-19835c8e6658\") " pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:00.356522 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.356482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7dn\" (UniqueName: \"kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn\") pod \"maas-api-key-cleanup-29724195-fh82w\" (UID: \"c4ccb445-58b0-4f7e-b8fc-19835c8e6658\") " pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:00.366357 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.366319 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7dn\" (UniqueName: \"kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn\") pod \"maas-api-key-cleanup-29724195-fh82w\" (UID: \"c4ccb445-58b0-4f7e-b8fc-19835c8e6658\") " pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:00.451925 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.451827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:00.594253 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.594218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-fh82w"] Jul 07 19:15:00.596472 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:15:00.596445 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ccb445_58b0_4f7e_b8fc_19835c8e6658.slice/crio-251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8 WatchSource:0}: Error finding container 251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8: Status 404 returned error can't find the container with id 251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8 Jul 07 19:15:00.622413 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.622367 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35fd4b3d-124e-47d3-abe8-7412264e4f3e" path="/var/lib/kubelet/pods/35fd4b3d-124e-47d3-abe8-7412264e4f3e/volumes" Jul 07 19:15:00.723061 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.722958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" event={"ID":"0ddb18ac-2704-484d-aca2-fea42d9f141e","Type":"ContainerStarted","Data":"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313"} Jul 07 19:15:00.723061 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.723033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" event={"ID":"0ddb18ac-2704-484d-aca2-fea42d9f141e","Type":"ContainerStarted","Data":"d85c4f1ba53d8792a89e53e98a43f173b22b3b33ef6d100a1a3fa58d8a630426"} Jul 07 19:15:00.724180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.724154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" event={"ID":"c4ccb445-58b0-4f7e-b8fc-19835c8e6658","Type":"ContainerStarted","Data":"251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8"} Jul 07 19:15:00.750152 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.750093 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" podStartSLOduration=1.7500701730000001 podStartE2EDuration="1.750070173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:00.748693344 +0000 UTC m=+1472.691316132" watchObservedRunningTime="2026-07-07 19:15:00.750070173 +0000 UTC m=+1472.692692962" Jul 07 19:15:00.877352 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:00.877305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:01.731168 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:01.731133 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4ccb445-58b0-4f7e-b8fc-19835c8e6658" containerID="fc701d8fd5e9a0e4bef9b9c4ecee0be0c3b33f37e85f1df12aad85ec7aeaf57b" exitCode=0 Jul 07 19:15:01.731565 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:01.731227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" event={"ID":"c4ccb445-58b0-4f7e-b8fc-19835c8e6658","Type":"ContainerDied","Data":"fc701d8fd5e9a0e4bef9b9c4ecee0be0c3b33f37e85f1df12aad85ec7aeaf57b"} Jul 07 19:15:01.883116 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:01.883032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:02.736174 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:02.736144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:02.737342 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:02.737312 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:02.894390 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:02.894352 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:02.984872 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:02.984838 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv7dn\" (UniqueName: \"kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn\") pod \"c4ccb445-58b0-4f7e-b8fc-19835c8e6658\" (UID: \"c4ccb445-58b0-4f7e-b8fc-19835c8e6658\") " Jul 07 19:15:02.987226 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:02.987150 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn" (OuterVolumeSpecName: "kube-api-access-wv7dn") pod "c4ccb445-58b0-4f7e-b8fc-19835c8e6658" (UID: "c4ccb445-58b0-4f7e-b8fc-19835c8e6658"). InnerVolumeSpecName "kube-api-access-wv7dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:15:03.085861 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:03.085818 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7dn\" (UniqueName: \"kubernetes.io/projected/c4ccb445-58b0-4f7e-b8fc-19835c8e6658-kube-api-access-wv7dn\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:03.230284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:03.230236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:15:03.742092 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:03.742062 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" Jul 07 19:15:03.742092 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:03.742080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-fh82w" event={"ID":"c4ccb445-58b0-4f7e-b8fc-19835c8e6658","Type":"ContainerDied","Data":"251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8"} Jul 07 19:15:03.742550 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:03.742138 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="251a91f37e487beb02f78e6c51c9e01902dc744208d338ff175a081c65db02f8" Jul 07 19:15:05.302348 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:05.302302 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:15:05.753175 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:05.753066 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" podUID="0ddb18ac-2704-484d-aca2-fea42d9f141e" containerName="istio-proxy" containerID="cri-o://4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313" gracePeriod=30 Jul 07 19:15:11.603753 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.603722 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:11.692807 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692713 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.692807 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692769 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692923 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.692982 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693307 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693048 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693307 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693075 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc57b\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b\") pod \"0ddb18ac-2704-484d-aca2-fea42d9f141e\" (UID: \"0ddb18ac-2704-484d-aca2-fea42d9f141e\") " Jul 07 19:15:11.693307 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693186 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:11.693307 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693203 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:15:11.693542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693469 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:11.693542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693517 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:11.693854 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693546 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.693854 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693568 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddb18ac-2704-484d-aca2-fea42d9f141e-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.694004 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.693984 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:11.695366 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.695332 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:11.695366 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.695344 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:15:11.695731 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.695701 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:15:11.695731 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.695711 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b" (OuterVolumeSpecName: "kube-api-access-rc57b") pod "0ddb18ac-2704-484d-aca2-fea42d9f141e" (UID: "0ddb18ac-2704-484d-aca2-fea42d9f141e"). InnerVolumeSpecName "kube-api-access-rc57b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:15:11.786200 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.786164 2564 generic.go:356] "Generic (PLEG): container finished" podID="0ddb18ac-2704-484d-aca2-fea42d9f141e" containerID="4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313" exitCode=0 Jul 07 19:15:11.786382 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.786211 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" Jul 07 19:15:11.786382 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.786251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" event={"ID":"0ddb18ac-2704-484d-aca2-fea42d9f141e","Type":"ContainerDied","Data":"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313"} Jul 07 19:15:11.786382 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.786317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk" event={"ID":"0ddb18ac-2704-484d-aca2-fea42d9f141e","Type":"ContainerDied","Data":"d85c4f1ba53d8792a89e53e98a43f173b22b3b33ef6d100a1a3fa58d8a630426"} Jul 07 19:15:11.786382 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.786344 2564 scope.go:117] "RemoveContainer" containerID="4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313" Jul 07 19:15:11.794293 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794257 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794300 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794314 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794327 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794338 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794352 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddb18ac-2704-484d-aca2-fea42d9f141e-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.794411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.794367 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc57b\" (UniqueName: \"kubernetes.io/projected/0ddb18ac-2704-484d-aca2-fea42d9f141e-kube-api-access-rc57b\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:15:11.795972 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.795952 2564 scope.go:117] "RemoveContainer" containerID="4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313" Jul 07 19:15:11.796275 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:15:11.796253 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313\": container with ID starting with 4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313 not found: ID does not exist" containerID="4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313" Jul 07 19:15:11.796327 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.796284 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313"} err="failed to get container status \"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313\": rpc error: code = NotFound desc = could not find container \"4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313\": container with ID starting with 4fac1b80f98afaa3d72654358cb6a749cb1013827bb839d24666774aa3f5e313 not found: ID does not exist" Jul 07 19:15:11.849587 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.849554 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:15:11.851308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.851282 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-914b2894-openshift-default-79bd5cb5f9-6cvhk"] Jul 07 19:15:11.906618 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:11.906581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:15:12.620876 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:15:12.620831 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ddb18ac-2704-484d-aca2-fea42d9f141e" path="/var/lib/kubelet/pods/0ddb18ac-2704-484d-aca2-fea42d9f141e/volumes" Jul 07 19:16:36.540314 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:36.540271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:16:44.032539 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:44.032503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:16:51.362168 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362119 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:16:51.362792 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362723 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ccb445-58b0-4f7e-b8fc-19835c8e6658" containerName="cleanup" Jul 07 19:16:51.362792 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362744 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ccb445-58b0-4f7e-b8fc-19835c8e6658" containerName="cleanup" Jul 07 19:16:51.362792 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362778 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ddb18ac-2704-484d-aca2-fea42d9f141e" containerName="istio-proxy" Jul 07 19:16:51.362792 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362786 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ddb18ac-2704-484d-aca2-fea42d9f141e" containerName="istio-proxy" Jul 07 19:16:51.363011 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362876 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ddb18ac-2704-484d-aca2-fea42d9f141e" containerName="istio-proxy" Jul 07 19:16:51.363011 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.362896 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ccb445-58b0-4f7e-b8fc-19835c8e6658" containerName="cleanup" Jul 07 19:16:51.366288 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.366262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.369151 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.369121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5d1e9a3e-openshift-default-dockercfg-c2mx9\"" Jul 07 19:16:51.398852 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.398821 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:16:51.456334 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456823 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456823 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.456823 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.456616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkll4\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.518951 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.518907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:16:51.557994 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.557951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.557994 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.557997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558259 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558259 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558378 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558255 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558378 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558378 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558521 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558521 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkll4\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558521 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.558898 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558871 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.559025 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.558997 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.559151 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.559028 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.559151 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.559064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.560775 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.560753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.561063 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.561049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.567284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.567256 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.567425 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.567306 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkll4\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4\") pod \"e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.677234 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.677137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:51.827258 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.827229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:16:51.829726 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:16:51.829696 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49508c5b_c940_4176_8834_6815d992056d.slice/crio-73b3450f9ed816c27bf51648d2e6aad5d2730c9720b029bab34b6812b84a69ca WatchSource:0}: Error finding container 73b3450f9ed816c27bf51648d2e6aad5d2730c9720b029bab34b6812b84a69ca: Status 404 returned error can't find the container with id 73b3450f9ed816c27bf51648d2e6aad5d2730c9720b029bab34b6812b84a69ca Jul 07 19:16:51.832015 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.831975 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:16:51.832085 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.832067 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:16:51.832130 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:51.832112 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:16:52.276621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:52.276583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerStarted","Data":"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed"} Jul 07 19:16:52.276820 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:52.276675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerStarted","Data":"73b3450f9ed816c27bf51648d2e6aad5d2730c9720b029bab34b6812b84a69ca"} Jul 07 19:16:52.319998 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:52.319932 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podStartSLOduration=1.319910881 podStartE2EDuration="1.319910881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:52.31971568 +0000 UTC m=+1584.262338468" watchObservedRunningTime="2026-07-07 19:16:52.319910881 +0000 UTC m=+1584.262533668" Jul 07 19:16:52.677614 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:52.677518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:52.682740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:52.682710 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:53.281914 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:53.281884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:53.283011 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:53.282990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:16:54.418002 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.417967 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:16:54.421707 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.421685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.424558 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.424527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:16:54.424558 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.424553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-5d1e9a3e-serving-cert\"" Jul 07 19:16:54.434175 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.434145 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:16:54.488946 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.488900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8s9\" (UniqueName: \"kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.489129 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.488960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.489129 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.489016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.590156 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.590116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8s9\" (UniqueName: \"kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.590156 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.590176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.590390 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.590343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.590791 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.590774 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.593065 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.593042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.601113 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.601085 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8s9\" (UniqueName: \"kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9\") pod \"maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.733275 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.733178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:54.882495 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:54.882466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:16:54.883991 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:16:54.883959 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863e0bbb_7425_4c93_97e5_721f1bd388ee.slice/crio-a6d5dfb2c696a80fcebd551ffe3510ab0039c658283fca84f5119381cf117f6a WatchSource:0}: Error finding container a6d5dfb2c696a80fcebd551ffe3510ab0039c658283fca84f5119381cf117f6a: Status 404 returned error can't find the container with id a6d5dfb2c696a80fcebd551ffe3510ab0039c658283fca84f5119381cf117f6a Jul 07 19:16:55.292547 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:55.292510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" event={"ID":"863e0bbb-7425-4c93-97e5-721f1bd388ee","Type":"ContainerStarted","Data":"a6d5dfb2c696a80fcebd551ffe3510ab0039c658283fca84f5119381cf117f6a"} Jul 07 19:16:56.300383 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:56.300347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" event={"ID":"863e0bbb-7425-4c93-97e5-721f1bd388ee","Type":"ContainerStarted","Data":"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01"} Jul 07 19:16:56.300856 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:56.300413 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:16:56.320074 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:56.320014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" podStartSLOduration=1.637999689 podStartE2EDuration="2.319994519s" totalImagesPullingTime="681.99483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:56.319143436 +0000 UTC m=+1588.261766223" watchObservedRunningTime="2026-07-07 19:16:56.319994519 +0000 UTC m=+1588.262617307" Jul 07 19:16:59.348487 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:16:59.348436 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:17:02.311237 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:02.311205 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:17:06.665226 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:06.665190 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:17:06.665789 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:06.665434 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" podUID="863e0bbb-7425-4c93-97e5-721f1bd388ee" containerName="maas-api" containerID="cri-o://d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01" gracePeriod=30 Jul 07 19:17:06.932761 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:06.932732 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:17:07.016006 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.015971 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr8s9\" (UniqueName: \"kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9\") pod \"863e0bbb-7425-4c93-97e5-721f1bd388ee\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " Jul 07 19:17:07.016203 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.016077 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls\") pod \"863e0bbb-7425-4c93-97e5-721f1bd388ee\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " Jul 07 19:17:07.016203 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.016172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca\") pod \"863e0bbb-7425-4c93-97e5-721f1bd388ee\" (UID: \"863e0bbb-7425-4c93-97e5-721f1bd388ee\") " Jul 07 19:17:07.016627 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.016591 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "863e0bbb-7425-4c93-97e5-721f1bd388ee" (UID: "863e0bbb-7425-4c93-97e5-721f1bd388ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:17:07.018420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.018389 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9" (OuterVolumeSpecName: "kube-api-access-nr8s9") pod "863e0bbb-7425-4c93-97e5-721f1bd388ee" (UID: "863e0bbb-7425-4c93-97e5-721f1bd388ee"). InnerVolumeSpecName "kube-api-access-nr8s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:07.018420 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.018395 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "863e0bbb-7425-4c93-97e5-721f1bd388ee" (UID: "863e0bbb-7425-4c93-97e5-721f1bd388ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:17:07.117139 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.117096 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863e0bbb-7425-4c93-97e5-721f1bd388ee-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.117139 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.117130 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8s9\" (UniqueName: \"kubernetes.io/projected/863e0bbb-7425-4c93-97e5-721f1bd388ee-kube-api-access-nr8s9\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.117139 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.117140 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863e0bbb-7425-4c93-97e5-721f1bd388ee-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.369874 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.369773 2564 generic.go:356] "Generic (PLEG): container finished" podID="863e0bbb-7425-4c93-97e5-721f1bd388ee" containerID="d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01" exitCode=0 Jul 07 19:17:07.369874 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.369815 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" Jul 07 19:17:07.370094 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.369874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" event={"ID":"863e0bbb-7425-4c93-97e5-721f1bd388ee","Type":"ContainerDied","Data":"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01"} Jul 07 19:17:07.370094 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.369928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr" event={"ID":"863e0bbb-7425-4c93-97e5-721f1bd388ee","Type":"ContainerDied","Data":"a6d5dfb2c696a80fcebd551ffe3510ab0039c658283fca84f5119381cf117f6a"} Jul 07 19:17:07.370094 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.369957 2564 scope.go:117] "RemoveContainer" containerID="d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01" Jul 07 19:17:07.371854 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.371828 2564 generic.go:356] "Generic (PLEG): container finished" podID="49508c5b-c940-4176-8834-6815d992056d" containerID="d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed" exitCode=0 Jul 07 19:17:07.372160 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.371902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerDied","Data":"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed"} Jul 07 19:17:07.372362 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.372346 2564 scope.go:117] "RemoveContainer" containerID="d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed" Jul 07 19:17:07.373465 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.373435 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:07.373575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.373510 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:07.373575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.373555 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:07.380058 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.380035 2564 scope.go:117] "RemoveContainer" containerID="d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01" Jul 07 19:17:07.380337 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:17:07.380320 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01\": container with ID starting with d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01 not found: ID does not exist" containerID="d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01" Jul 07 19:17:07.380406 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.380344 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01"} err="failed to get container status \"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01\": rpc error: code = NotFound desc = could not find container \"d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01\": container with ID starting with d9681c3f381307e3bac60a2cfaf20b2cf2436e2d0e8a13851006d2ed286f1f01 not found: ID does not exist" Jul 07 19:17:07.411468 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.411438 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:17:07.415361 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:07.415331 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-5d1e9a3e-5c5887bf7c-lbgqr"] Jul 07 19:17:08.282713 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.282626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:08.379190 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.379148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerStarted","Data":"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730"} Jul 07 19:17:08.624622 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.624515 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863e0bbb-7425-4c93-97e5-721f1bd388ee" path="/var/lib/kubelet/pods/863e0bbb-7425-4c93-97e5-721f1bd388ee/volumes" Jul 07 19:17:08.678387 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.678336 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:08.679558 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.679526 2564 patch_prober.go:28] interesting pod/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 07 19:17:08.679692 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:08.679580 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 07 19:17:09.678042 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:09.678000 2564 patch_prober.go:28] interesting pod/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 07 19:17:09.678437 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:09.678060 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 07 19:17:10.039611 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:10.039568 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:17:10.678284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:10.678241 2564 patch_prober.go:28] interesting pod/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 07 19:17:10.678726 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:10.678316 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 07 19:17:11.677622 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:11.677578 2564 patch_prober.go:28] interesting pod/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 07 19:17:11.677840 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:11.677682 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 07 19:17:12.681785 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:12.681748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:12.682301 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:12.682285 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:12.683148 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:12.683129 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:13.534359 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:13.534322 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:17:14.410023 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:14.409985 2564 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" secret="" err="secret \"e2e-mt-5d1e9a3e-openshift-default-dockercfg-c2mx9\" not found" Jul 07 19:17:15.414963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:15.414901 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" containerID="cri-o://779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730" gracePeriod=30 Jul 07 19:17:20.678764 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.678695 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:20.751723 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.751920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.751920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751794 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.751920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.751920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.751920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.752176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.751933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.752176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.752176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkll4\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4\") pod \"49508c5b-c940-4176-8834-6815d992056d\" (UID: \"49508c5b-c940-4176-8834-6815d992056d\") " Jul 07 19:17:20.752754 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752431 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:20.752754 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752479 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:17:20.752754 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752683 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:20.752754 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752743 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:20.753098 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.752869 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data" (OuterVolumeSpecName: "istio-data") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:20.754416 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.754391 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token" (OuterVolumeSpecName: "istio-token") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:20.754549 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.754411 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:17:20.754637 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.754615 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:20.754850 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.754833 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4" (OuterVolumeSpecName: "kube-api-access-fkll4") pod "49508c5b-c940-4176-8834-6815d992056d" (UID: "49508c5b-c940-4176-8834-6815d992056d"). InnerVolumeSpecName "kube-api-access-fkll4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:20.853044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853008 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853041 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853051 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkll4\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-kube-api-access-fkll4\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853059 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853069 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853077 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49508c5b-c940-4176-8834-6815d992056d-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853084 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49508c5b-c940-4176-8834-6815d992056d-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853091 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49508c5b-c940-4176-8834-6815d992056d-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:20.853279 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:20.853099 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49508c5b-c940-4176-8834-6815d992056d-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:17:21.447858 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.447825 2564 generic.go:356] "Generic (PLEG): container finished" podID="49508c5b-c940-4176-8834-6815d992056d" containerID="779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730" exitCode=0 Jul 07 19:17:21.448042 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.447879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" Jul 07 19:17:21.448042 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.447916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerDied","Data":"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730"} Jul 07 19:17:21.448042 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.447977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l" event={"ID":"49508c5b-c940-4176-8834-6815d992056d","Type":"ContainerDied","Data":"73b3450f9ed816c27bf51648d2e6aad5d2730c9720b029bab34b6812b84a69ca"} Jul 07 19:17:21.448042 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.448007 2564 scope.go:117] "RemoveContainer" containerID="779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730" Jul 07 19:17:21.457734 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.457714 2564 scope.go:117] "RemoveContainer" containerID="d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed" Jul 07 19:17:21.466333 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.466311 2564 scope.go:117] "RemoveContainer" containerID="779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730" Jul 07 19:17:21.466681 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:17:21.466661 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730\": container with ID starting with 779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730 not found: ID does not exist" containerID="779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730" Jul 07 19:17:21.466724 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.466693 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730"} err="failed to get container status \"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730\": rpc error: code = NotFound desc = could not find container \"779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730\": container with ID starting with 779c04682143817d22e51ec815e5af446afa4ff1a90f326a53fd654decc60730 not found: ID does not exist" Jul 07 19:17:21.466724 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.466711 2564 scope.go:117] "RemoveContainer" containerID="d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed" Jul 07 19:17:21.466989 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:17:21.466969 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed\": container with ID starting with d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed not found: ID does not exist" containerID="d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed" Jul 07 19:17:21.467055 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.466995 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed"} err="failed to get container status \"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed\": rpc error: code = NotFound desc = could not find container \"d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed\": container with ID starting with d97c2b3de630a7f6340ff988efe49e10429e1fa9dd77b026f854f6c2a263a1ed not found: ID does not exist" Jul 07 19:17:21.478222 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.478187 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:17:21.484275 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:21.484245 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5d1e9a3e-openshift-default-7cbd64bdc9-hlb2l"] Jul 07 19:17:22.621124 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:22.621089 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49508c5b-c940-4176-8834-6815d992056d" path="/var/lib/kubelet/pods/49508c5b-c940-4176-8834-6815d992056d/volumes" Jul 07 19:17:34.277032 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.276987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277433 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277448 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277465 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863e0bbb-7425-4c93-97e5-721f1bd388ee" containerName="maas-api" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="863e0bbb-7425-4c93-97e5-721f1bd388ee" containerName="maas-api" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277535 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="863e0bbb-7425-4c93-97e5-721f1bd388ee" containerName="maas-api" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277546 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:34.277630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.277553 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:34.280513 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.280495 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.283113 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.283082 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fec6862b-openshift-default-dockercfg-shlws\"" Jul 07 19:17:34.298776 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.298738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:17:34.379074 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379074 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkwr\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379284 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.379454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.379336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480045 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.479980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480045 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkwr\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480536 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480536 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480682 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480682 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480602 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480797 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480643 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.480797 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.480764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.481044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.481025 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.482735 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.482719 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.482819 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.482804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.493878 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.493848 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.495098 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.495068 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkwr\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr\") pod \"e2e-mt-fec6862b-openshift-default-7577774955-ztv59\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.591419 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.591311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:34.745543 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.745516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:17:34.748194 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:17:34.748170 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa03a4c7_c53c_450d_bf1c_29360e02070c.slice/crio-aba4912a765453f0073e08bd224f981d599db99cd1db6d2b67f8608d39de9c37 WatchSource:0}: Error finding container aba4912a765453f0073e08bd224f981d599db99cd1db6d2b67f8608d39de9c37: Status 404 returned error can't find the container with id aba4912a765453f0073e08bd224f981d599db99cd1db6d2b67f8608d39de9c37 Jul 07 19:17:34.750457 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.750420 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:34.750548 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.750495 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:34.750548 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:34.750531 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:35.516238 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:35.516204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerStarted","Data":"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d"} Jul 07 19:17:35.516673 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:35.516262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerStarted","Data":"aba4912a765453f0073e08bd224f981d599db99cd1db6d2b67f8608d39de9c37"} Jul 07 19:17:35.541908 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:35.541836 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podStartSLOduration=1.54181424 podStartE2EDuration="1.54181424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:35.538886618 +0000 UTC m=+1627.481509427" watchObservedRunningTime="2026-07-07 19:17:35.54181424 +0000 UTC m=+1627.484437029" Jul 07 19:17:35.591559 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:35.591521 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:35.596580 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:35.596554 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:36.521621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:36.521587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:36.522562 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:36.522542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:37.734620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.734578 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:17:37.735200 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.735179 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:37.735247 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.735205 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49508c5b-c940-4176-8834-6815d992056d" containerName="istio-proxy" Jul 07 19:17:37.737920 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.737896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.740469 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.740443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-fec6862b-serving-cert\"" Jul 07 19:17:37.740684 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.740665 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:17:37.746282 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.746244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:17:37.814732 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.814588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw96\" (UniqueName: \"kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.814732 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.814685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.814934 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.814813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.916028 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.915992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.916228 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.916070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw96\" (UniqueName: \"kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.916228 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.916107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.916703 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.916643 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.919003 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.918975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:37.927298 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:37.927268 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw96\" (UniqueName: \"kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96\") pod \"maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:38.051557 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:38.051513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:38.196086 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:38.196058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:17:38.197778 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:17:38.197747 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a23be8e_1b5e_4c4b_a985_35779e686b7b.slice/crio-0758804a2cb0b95a9f0861ff471e820cb51f3cd9efdfbcc16146ce0a12f11d18 WatchSource:0}: Error finding container 0758804a2cb0b95a9f0861ff471e820cb51f3cd9efdfbcc16146ce0a12f11d18: Status 404 returned error can't find the container with id 0758804a2cb0b95a9f0861ff471e820cb51f3cd9efdfbcc16146ce0a12f11d18 Jul 07 19:17:38.533171 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:38.533127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" event={"ID":"1a23be8e-1b5e-4c4b-a985-35779e686b7b","Type":"ContainerStarted","Data":"0758804a2cb0b95a9f0861ff471e820cb51f3cd9efdfbcc16146ce0a12f11d18"} Jul 07 19:17:39.539559 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:39.539521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" event={"ID":"1a23be8e-1b5e-4c4b-a985-35779e686b7b","Type":"ContainerStarted","Data":"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882"} Jul 07 19:17:39.540008 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:39.539624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:39.559072 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:39.559013 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" podStartSLOduration=2.039058944 podStartE2EDuration="2.558997041s" totalImagesPullingTime="519.938097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:39.556879602 +0000 UTC m=+1631.499502389" watchObservedRunningTime="2026-07-07 19:17:39.558997041 +0000 UTC m=+1631.501619828" Jul 07 19:17:42.034971 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:42.034930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:17:45.550436 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:45.550404 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:17:50.600621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.600584 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerID="2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d" exitCode=0 Jul 07 19:17:50.601078 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.600645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerDied","Data":"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d"} Jul 07 19:17:50.601123 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.601107 2564 scope.go:117] "RemoveContainer" containerID="2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d" Jul 07 19:17:50.602207 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.602174 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:50.602308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.602244 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:50.602308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:50.602277 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:51.522321 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:51.522276 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:51.607515 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:51.607479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerStarted","Data":"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06"} Jul 07 19:17:52.591933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:52.591888 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:52.593159 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:52.593130 2564 patch_prober.go:28] interesting pod/e2e-mt-fec6862b-openshift-default-7577774955-ztv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 07 19:17:52.593329 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:52.593191 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 07 19:17:53.337805 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:53.337737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:17:53.592402 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:53.592307 2564 patch_prober.go:28] interesting pod/e2e-mt-fec6862b-openshift-default-7577774955-ztv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 07 19:17:53.592402 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:53.592373 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 07 19:17:54.592815 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:54.592778 2564 patch_prober.go:28] interesting pod/e2e-mt-fec6862b-openshift-default-7577774955-ztv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 07 19:17:54.593304 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:54.593274 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 07 19:17:55.666998 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:55.666945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:55.667557 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:55.667537 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:17:55.668591 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:17:55.668561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:18:07.332271 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:07.332238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:18:10.368577 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.368541 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:18:10.369137 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.368771 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerName="maas-api" containerID="cri-o://df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882" gracePeriod=30 Jul 07 19:18:10.639148 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.639075 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:18:10.708557 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.708517 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerID="df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882" exitCode=0 Jul 07 19:18:10.708784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.708576 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" Jul 07 19:18:10.708784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.708642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" event={"ID":"1a23be8e-1b5e-4c4b-a985-35779e686b7b","Type":"ContainerDied","Data":"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882"} Jul 07 19:18:10.708784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.708744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" event={"ID":"1a23be8e-1b5e-4c4b-a985-35779e686b7b","Type":"ContainerDied","Data":"0758804a2cb0b95a9f0861ff471e820cb51f3cd9efdfbcc16146ce0a12f11d18"} Jul 07 19:18:10.708784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.708766 2564 scope.go:117] "RemoveContainer" containerID="df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882" Jul 07 19:18:10.717966 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.717948 2564 scope.go:117] "RemoveContainer" containerID="df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882" Jul 07 19:18:10.718291 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:18:10.718266 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882\": container with ID starting with df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882 not found: ID does not exist" containerID="df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882" Jul 07 19:18:10.718387 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.718308 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882"} err="failed to get container status \"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882\": rpc error: code = NotFound desc = could not find container \"df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882\": container with ID starting with df20c15f142b8d1d8d74848e8032d55265544b99a2c904db4c00b3b3ce1c5882 not found: ID does not exist" Jul 07 19:18:10.771218 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.771177 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls\") pod \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " Jul 07 19:18:10.771389 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.771227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkw96\" (UniqueName: \"kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96\") pod \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " Jul 07 19:18:10.771389 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.771271 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca\") pod \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\" (UID: \"1a23be8e-1b5e-4c4b-a985-35779e686b7b\") " Jul 07 19:18:10.771695 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.771671 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a23be8e-1b5e-4c4b-a985-35779e686b7b" (UID: "1a23be8e-1b5e-4c4b-a985-35779e686b7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:18:10.773668 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.773630 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a23be8e-1b5e-4c4b-a985-35779e686b7b" (UID: "1a23be8e-1b5e-4c4b-a985-35779e686b7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:18:10.773668 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.773634 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96" (OuterVolumeSpecName: "kube-api-access-rkw96") pod "1a23be8e-1b5e-4c4b-a985-35779e686b7b" (UID: "1a23be8e-1b5e-4c4b-a985-35779e686b7b"). InnerVolumeSpecName "kube-api-access-rkw96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:18:10.873064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.873025 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a23be8e-1b5e-4c4b-a985-35779e686b7b-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:18:10.873064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.873057 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkw96\" (UniqueName: \"kubernetes.io/projected/1a23be8e-1b5e-4c4b-a985-35779e686b7b-kube-api-access-rkw96\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:18:10.873064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:10.873068 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a23be8e-1b5e-4c4b-a985-35779e686b7b-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:18:11.033951 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:11.033913 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:18:11.039147 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:11.039116 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn"] Jul 07 19:18:12.621109 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:12.621074 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" path="/var/lib/kubelet/pods/1a23be8e-1b5e-4c4b-a985-35779e686b7b/volumes" Jul 07 19:18:13.546061 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:13.545988 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-fec6862b-59987dfc6d-5b6xn" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.40:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 07 19:18:21.458949 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:18:21.458910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:19:04.527892 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:04.527797 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:19:05.555076 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:05.555038 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:19:05.555481 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:05.555270 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" containerID="cri-o://b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06" gracePeriod=30 Jul 07 19:19:10.668971 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.668934 2564 patch_prober.go:28] interesting pod/e2e-mt-fec6862b-openshift-default-7577774955-ztv59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 07 19:19:10.669355 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.669002 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 07 19:19:10.819282 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.819208 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:19:10.848609 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.848818 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.848818 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848692 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.848818 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.848818 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848768 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.848818 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.849081 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.849081 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848890 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.849081 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.848922 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnkwr\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr\") pod \"fa03a4c7-c53c-450d-bf1c-29360e02070c\" (UID: \"fa03a4c7-c53c-450d-bf1c-29360e02070c\") " Jul 07 19:19:10.849289 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.849235 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:19:10.849376 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.849279 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:19:10.849504 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.849467 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:19:10.849586 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.849524 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:19:10.850319 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.850287 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:19:10.851310 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.851289 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr" (OuterVolumeSpecName: "kube-api-access-fnkwr") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "kube-api-access-fnkwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:19:10.851439 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.851420 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:19:10.851526 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.851507 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:19:10.851565 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.851509 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa03a4c7-c53c-450d-bf1c-29360e02070c" (UID: "fa03a4c7-c53c-450d-bf1c-29360e02070c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:19:10.950477 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950428 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950477 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950465 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950477 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950473 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950477 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950481 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950477 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950489 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950820 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950497 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa03a4c7-c53c-450d-bf1c-29360e02070c-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950820 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950506 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950820 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950512 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa03a4c7-c53c-450d-bf1c-29360e02070c-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:10.950820 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:10.950544 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkwr\" (UniqueName: \"kubernetes.io/projected/fa03a4c7-c53c-450d-bf1c-29360e02070c-kube-api-access-fnkwr\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:19:11.005612 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.005577 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerID="b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06" exitCode=0 Jul 07 19:19:11.005798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.005629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerDied","Data":"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06"} Jul 07 19:19:11.005798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.005664 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" Jul 07 19:19:11.005798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.005729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59" event={"ID":"fa03a4c7-c53c-450d-bf1c-29360e02070c","Type":"ContainerDied","Data":"aba4912a765453f0073e08bd224f981d599db99cd1db6d2b67f8608d39de9c37"} Jul 07 19:19:11.005798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.005752 2564 scope.go:117] "RemoveContainer" containerID="b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06" Jul 07 19:19:11.014851 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.014833 2564 scope.go:117] "RemoveContainer" containerID="2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d" Jul 07 19:19:11.023563 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.023543 2564 scope.go:117] "RemoveContainer" containerID="b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06" Jul 07 19:19:11.023892 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:19:11.023867 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06\": container with ID starting with b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06 not found: ID does not exist" containerID="b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06" Jul 07 19:19:11.024003 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.023899 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06"} err="failed to get container status \"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06\": rpc error: code = NotFound desc = could not find container \"b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06\": container with ID starting with b8ee36cd0411ce2234ec27fa4f113c063ffcb833f02b9148082ee614a4004f06 not found: ID does not exist" Jul 07 19:19:11.024003 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.023918 2564 scope.go:117] "RemoveContainer" containerID="2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d" Jul 07 19:19:11.024155 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:19:11.024138 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d\": container with ID starting with 2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d not found: ID does not exist" containerID="2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d" Jul 07 19:19:11.024199 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.024160 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d"} err="failed to get container status \"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d\": rpc error: code = NotFound desc = could not find container \"2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d\": container with ID starting with 2f3ceb5e404388f2fdbabb9d87e0d18c26ae90d95202cbf46d4829dd2f78a24d not found: ID does not exist" Jul 07 19:19:11.031251 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.031218 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:19:11.038572 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:11.038536 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fec6862b-openshift-default-7577774955-ztv59"] Jul 07 19:19:12.620718 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:12.620683 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" path="/var/lib/kubelet/pods/fa03a4c7-c53c-450d-bf1c-29360e02070c/volumes" Jul 07 19:19:16.244246 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:16.244200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:19:24.036467 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:19:24.036429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:20:01.654608 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.654549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:20:01.655407 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655382 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:01.655519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655412 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:01.655519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655424 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:01.655519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655433 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:01.655519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655455 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerName="maas-api" Jul 07 19:20:01.655519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655466 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerName="maas-api" Jul 07 19:20:01.655791 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655679 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:01.655791 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.655696 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a23be8e-1b5e-4c4b-a985-35779e686b7b" containerName="maas-api" Jul 07 19:20:01.659062 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.659037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.661625 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.661605 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4e0129-openshift-default-dockercfg-sx9l2\"" Jul 07 19:20:01.668957 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.668912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:20:01.729630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.729590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.729864 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.729762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.729942 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.729887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kf95\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730095 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730191 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730191 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730327 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730811 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.730943 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.730850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831699 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831699 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831917 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831917 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831996 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.831996 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.831963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kf95\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832074 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832074 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832179 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832061 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832179 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832276 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832276 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832368 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.832751 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.832732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.834357 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.834324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.834560 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.834540 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.841164 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.841133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.841318 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.841299 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kf95\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95\") pod \"e2e-inf-a-4e0129-openshift-default-689dd64867-q577g\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:01.971184 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:01.971083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:02.119474 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.119447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:20:02.121866 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:20:02.121829 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027c6f3c_4c8e_4d30_9bc2_4297c6d7e667.slice/crio-f6dae95fe9b5deb7979aaa4dec134318e0f9ec7a039167c5da28f234872c9ed4 WatchSource:0}: Error finding container f6dae95fe9b5deb7979aaa4dec134318e0f9ec7a039167c5da28f234872c9ed4: Status 404 returned error can't find the container with id f6dae95fe9b5deb7979aaa4dec134318e0f9ec7a039167c5da28f234872c9ed4 Jul 07 19:20:02.124044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.124007 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:02.124143 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.124082 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:02.124143 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.124110 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:02.249178 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.249090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerStarted","Data":"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661"} Jul 07 19:20:02.249178 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.249157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerStarted","Data":"f6dae95fe9b5deb7979aaa4dec134318e0f9ec7a039167c5da28f234872c9ed4"} Jul 07 19:20:02.271994 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.271926 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podStartSLOduration=1.271903843 podStartE2EDuration="1.271903843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:20:02.270824729 +0000 UTC m=+1774.213447518" watchObservedRunningTime="2026-07-07 19:20:02.271903843 +0000 UTC m=+1774.214526631" Jul 07 19:20:02.971437 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:02.971395 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:03.977300 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:03.977270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:04.259205 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.259115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:04.260216 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.260198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:04.440507 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.440473 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:20:04.440961 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.440946 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa03a4c7-c53c-450d-bf1c-29360e02070c" containerName="istio-proxy" Jul 07 19:20:04.443043 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.443023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.445261 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.445241 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-4e0129-serving-cert\"" Jul 07 19:20:04.445538 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.445521 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:20:04.456760 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.456729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:20:04.558039 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.557995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnts\" (UniqueName: \"kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.558264 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.558055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.558264 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.558193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.658975 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.658940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.659175 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.659038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnts\" (UniqueName: \"kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.659175 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.659094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.659784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.659762 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.661590 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.661570 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.670194 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.670164 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnts\" (UniqueName: \"kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts\") pod \"maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.755299 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.755254 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:04.903264 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.903239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:20:04.905338 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:20:04.905297 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b399460_21f8_49bf_86ba_f87fca539d6e.slice/crio-d68d45bfeee5bb1adc503eeba0281de64a7653a7190ed627b07f286d79482c99 WatchSource:0}: Error finding container d68d45bfeee5bb1adc503eeba0281de64a7653a7190ed627b07f286d79482c99: Status 404 returned error can't find the container with id d68d45bfeee5bb1adc503eeba0281de64a7653a7190ed627b07f286d79482c99 Jul 07 19:20:04.907254 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:04.907236 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:20:05.265784 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:05.265699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" event={"ID":"3b399460-21f8-49bf-86ba-f87fca539d6e","Type":"ContainerStarted","Data":"d68d45bfeee5bb1adc503eeba0281de64a7653a7190ed627b07f286d79482c99"} Jul 07 19:20:06.278870 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:06.278822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" event={"ID":"3b399460-21f8-49bf-86ba-f87fca539d6e","Type":"ContainerStarted","Data":"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95"} Jul 07 19:20:06.301035 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:06.300969 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" podStartSLOduration=1.930272804 podStartE2EDuration="2.30095248s" totalImagesPullingTime="370.679676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:20:06.299583108 +0000 UTC m=+1778.242205896" watchObservedRunningTime="2026-07-07 19:20:06.30095248 +0000 UTC m=+1778.243575268" Jul 07 19:20:07.284336 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:07.284302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:09.377422 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.377387 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:20:09.380450 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.380427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.383447 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.383427 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2e4d95-openshift-default-dockercfg-r9ggp\"" Jul 07 19:20:09.399790 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.399756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:20:09.509166 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509690 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.509690 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.509464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxrk\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610573 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610573 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610573 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610573 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610573 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610873 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxrk\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610873 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610873 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610873 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610841 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.610873 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.610861 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.611128 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.611093 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.611391 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.611372 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.613454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.613423 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.613567 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.613529 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.622071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.622048 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxrk\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.622215 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.622170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token\") pod \"e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.691490 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.691389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:09.845811 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.845781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:20:09.847666 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:20:09.847625 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b80cc2_ff20_4451_8de4_672e549fa948.slice/crio-7d985038ff9c6dc475800d9ca1e9d1e0c5223e8eaba6cd59ab0e7a82483e34f9 WatchSource:0}: Error finding container 7d985038ff9c6dc475800d9ca1e9d1e0c5223e8eaba6cd59ab0e7a82483e34f9: Status 404 returned error can't find the container with id 7d985038ff9c6dc475800d9ca1e9d1e0c5223e8eaba6cd59ab0e7a82483e34f9 Jul 07 19:20:09.849838 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.849805 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:09.849912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.849874 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:09.849912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:09.849903 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:10.300404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:10.300366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerStarted","Data":"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621"} Jul 07 19:20:10.300614 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:10.300424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerStarted","Data":"7d985038ff9c6dc475800d9ca1e9d1e0c5223e8eaba6cd59ab0e7a82483e34f9"} Jul 07 19:20:10.329670 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:10.329599 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podStartSLOduration=1.329581178 podStartE2EDuration="1.329581178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:20:10.32718399 +0000 UTC m=+1782.269806816" watchObservedRunningTime="2026-07-07 19:20:10.329581178 +0000 UTC m=+1782.272203965" Jul 07 19:20:10.692591 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:10.692488 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:10.697774 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:10.697742 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:11.306036 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:11.306003 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:11.307018 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:11.306995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:12.027980 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.027942 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:20:12.324426 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.324338 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:20:12.327300 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.327277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.329717 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.329687 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-2e4d95-serving-cert\"" Jul 07 19:20:12.340812 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.340775 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:20:12.443643 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.443177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.443643 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.443249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vqh\" (UniqueName: \"kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.443643 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.443349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.544122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.544087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.544330 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.544163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.544330 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.544202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vqh\" (UniqueName: \"kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.544802 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.544780 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.546808 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.546787 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.553416 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.553390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vqh\" (UniqueName: \"kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh\") pod \"maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.639698 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.639586 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:12.804925 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:12.804861 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:20:12.807418 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:20:12.807392 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10412ebf_7e51_446c_baa6_5c25d08383f2.slice/crio-db384b26ff3e73746c5b85c0eec0911638a0065ccbec9b050c729723ca59bde0 WatchSource:0}: Error finding container db384b26ff3e73746c5b85c0eec0911638a0065ccbec9b050c729723ca59bde0: Status 404 returned error can't find the container with id db384b26ff3e73746c5b85c0eec0911638a0065ccbec9b050c729723ca59bde0 Jul 07 19:20:13.295492 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:13.295462 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:20:13.332675 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:13.331927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" event={"ID":"10412ebf-7e51-446c-baa6-5c25d08383f2","Type":"ContainerStarted","Data":"db384b26ff3e73746c5b85c0eec0911638a0065ccbec9b050c729723ca59bde0"} Jul 07 19:20:14.340835 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:14.340793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" event={"ID":"10412ebf-7e51-446c-baa6-5c25d08383f2","Type":"ContainerStarted","Data":"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676"} Jul 07 19:20:14.341237 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:14.340883 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:14.360064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:14.359990 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" podStartSLOduration=1.8609226319999999 podStartE2EDuration="2.359969728s" totalImagesPullingTime="499.047096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:20:14.359685857 +0000 UTC m=+1786.302308643" watchObservedRunningTime="2026-07-07 19:20:14.359969728 +0000 UTC m=+1786.302592515" Jul 07 19:20:20.050341 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.050302 2564 patch_prober.go:28] interesting pod/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:46938->10.134.0.41:15021: read: connection reset by peer" start-of-body= Jul 07 19:20:20.050814 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.050386 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:46938->10.134.0.41:15021: read: connection reset by peer" Jul 07 19:20:20.352014 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.351932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:20:20.374212 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.374179 2564 generic.go:356] "Generic (PLEG): container finished" podID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerID="859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661" exitCode=0 Jul 07 19:20:20.374379 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.374254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerDied","Data":"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661"} Jul 07 19:20:20.374698 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.374684 2564 scope.go:117] "RemoveContainer" containerID="859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661" Jul 07 19:20:20.375994 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.375952 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:20.376117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.376054 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:20.376117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:20.376100 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:21.383860 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:21.383822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerStarted","Data":"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad"} Jul 07 19:20:21.971306 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:21.971261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:21.971642 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:21.971620 2564 patch_prober.go:28] interesting pod/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:20:21.971723 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:21.971698 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 19:20:22.971850 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:22.971803 2564 patch_prober.go:28] interesting pod/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:20:22.972317 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:22.971862 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 19:20:23.315001 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:23.314956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:20:23.971845 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:23.971793 2564 patch_prober.go:28] interesting pod/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:20:23.972044 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:23.971912 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 19:20:24.971781 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:24.971741 2564 patch_prober.go:28] interesting pod/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:20:24.971963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:24.971804 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 19:20:25.975130 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:25.975098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:25.975543 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:25.975481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:25.976155 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:25.976137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:20:26.307995 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.307951 2564 patch_prober.go:28] interesting pod/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:20:26.308237 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.308033 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 19:20:26.416329 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.416250 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerID="b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621" exitCode=0 Jul 07 19:20:26.416485 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.416321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerDied","Data":"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621"} Jul 07 19:20:26.416906 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.416891 2564 scope.go:117] "RemoveContainer" containerID="b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621" Jul 07 19:20:26.417890 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.417863 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:26.417944 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.417933 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:26.417981 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:26.417966 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:20:27.429113 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:27.429056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerStarted","Data":"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf"} Jul 07 19:20:27.692303 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:27.692209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:27.692517 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:27.692492 2564 patch_prober.go:28] interesting pod/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:20:27.692565 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:27.692546 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 19:20:28.692374 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:28.692338 2564 patch_prober.go:28] interesting pod/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:20:28.692780 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:28.692399 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 19:20:29.691937 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:29.691900 2564 patch_prober.go:28] interesting pod/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:20:29.692123 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:29.691975 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 19:20:30.691978 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:30.691934 2564 patch_prober.go:28] interesting pod/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:20:30.692367 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:30.692008 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 19:20:31.696828 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:31.696789 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:31.697296 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:31.697052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:31.697923 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:31.697899 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:20:35.944508 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:35.944470 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:20:45.422220 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:45.422168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:20:57.026115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:20:57.026079 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:21:17.411672 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.411618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:21:17.412237 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.411902 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" podUID="3b399460-21f8-49bf-86ba-f87fca539d6e" containerName="maas-api" containerID="cri-o://d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95" gracePeriod=30 Jul 07 19:21:17.670472 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.670408 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:21:17.704052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.704019 2564 generic.go:356] "Generic (PLEG): container finished" podID="3b399460-21f8-49bf-86ba-f87fca539d6e" containerID="d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95" exitCode=0 Jul 07 19:21:17.704232 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.704085 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" Jul 07 19:21:17.704232 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.704093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" event={"ID":"3b399460-21f8-49bf-86ba-f87fca539d6e","Type":"ContainerDied","Data":"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95"} Jul 07 19:21:17.704232 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.704149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh" event={"ID":"3b399460-21f8-49bf-86ba-f87fca539d6e","Type":"ContainerDied","Data":"d68d45bfeee5bb1adc503eeba0281de64a7653a7190ed627b07f286d79482c99"} Jul 07 19:21:17.704232 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.704171 2564 scope.go:117] "RemoveContainer" containerID="d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95" Jul 07 19:21:17.714645 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.714619 2564 scope.go:117] "RemoveContainer" containerID="d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95" Jul 07 19:21:17.715019 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:21:17.714996 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95\": container with ID starting with d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95 not found: ID does not exist" containerID="d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95" Jul 07 19:21:17.715079 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.715035 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95"} err="failed to get container status \"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95\": rpc error: code = NotFound desc = could not find container \"d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95\": container with ID starting with d6175b959ebc436932ad680a2cfef57228a96591041aaf6ceeea85a417d35f95 not found: ID does not exist" Jul 07 19:21:17.745113 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.745066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls\") pod \"3b399460-21f8-49bf-86ba-f87fca539d6e\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " Jul 07 19:21:17.745273 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.745147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmnts\" (UniqueName: \"kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts\") pod \"3b399460-21f8-49bf-86ba-f87fca539d6e\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " Jul 07 19:21:17.745273 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.745243 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca\") pod \"3b399460-21f8-49bf-86ba-f87fca539d6e\" (UID: \"3b399460-21f8-49bf-86ba-f87fca539d6e\") " Jul 07 19:21:17.745621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.745597 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b399460-21f8-49bf-86ba-f87fca539d6e" (UID: "3b399460-21f8-49bf-86ba-f87fca539d6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:21:17.747518 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.747485 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b399460-21f8-49bf-86ba-f87fca539d6e" (UID: "3b399460-21f8-49bf-86ba-f87fca539d6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:21:17.747635 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.747524 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts" (OuterVolumeSpecName: "kube-api-access-rmnts") pod "3b399460-21f8-49bf-86ba-f87fca539d6e" (UID: "3b399460-21f8-49bf-86ba-f87fca539d6e"). InnerVolumeSpecName "kube-api-access-rmnts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:21:17.846176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.846138 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b399460-21f8-49bf-86ba-f87fca539d6e-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:21:17.846176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.846172 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b399460-21f8-49bf-86ba-f87fca539d6e-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:21:17.846176 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:17.846184 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnts\" (UniqueName: \"kubernetes.io/projected/3b399460-21f8-49bf-86ba-f87fca539d6e-kube-api-access-rmnts\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:21:18.026270 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:18.026235 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:21:18.031310 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:18.031281 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-4e0129-8655ddd689-l9dlh"] Jul 07 19:21:18.620609 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:18.620571 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b399460-21f8-49bf-86ba-f87fca539d6e" path="/var/lib/kubelet/pods/3b399460-21f8-49bf-86ba-f87fca539d6e/volumes" Jul 07 19:21:20.128620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:20.128579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:21:59.610217 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:59.610123 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:21:59.610739 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:21:59.610425 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" containerID="cri-o://c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad" gracePeriod=30 Jul 07 19:22:00.924505 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:00.924464 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:22:00.925055 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:00.924744 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" podUID="10412ebf-7e51-446c-baa6-5c25d08383f2" containerName="maas-api" containerID="cri-o://98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676" gracePeriod=30 Jul 07 19:22:01.179783 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.179705 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:22:01.287041 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.286994 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4vqh\" (UniqueName: \"kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh\") pod \"10412ebf-7e51-446c-baa6-5c25d08383f2\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " Jul 07 19:22:01.287041 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.287043 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls\") pod \"10412ebf-7e51-446c-baa6-5c25d08383f2\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " Jul 07 19:22:01.287256 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.287127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca\") pod \"10412ebf-7e51-446c-baa6-5c25d08383f2\" (UID: \"10412ebf-7e51-446c-baa6-5c25d08383f2\") " Jul 07 19:22:01.287609 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.287567 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10412ebf-7e51-446c-baa6-5c25d08383f2" (UID: "10412ebf-7e51-446c-baa6-5c25d08383f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:01.289402 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.289351 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10412ebf-7e51-446c-baa6-5c25d08383f2" (UID: "10412ebf-7e51-446c-baa6-5c25d08383f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:22:01.289524 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.289418 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh" (OuterVolumeSpecName: "kube-api-access-d4vqh") pod "10412ebf-7e51-446c-baa6-5c25d08383f2" (UID: "10412ebf-7e51-446c-baa6-5c25d08383f2"). InnerVolumeSpecName "kube-api-access-d4vqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:01.388475 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.388427 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10412ebf-7e51-446c-baa6-5c25d08383f2-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:01.388475 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.388466 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vqh\" (UniqueName: \"kubernetes.io/projected/10412ebf-7e51-446c-baa6-5c25d08383f2-kube-api-access-d4vqh\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:01.388475 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.388476 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10412ebf-7e51-446c-baa6-5c25d08383f2-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:01.924488 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.924455 2564 generic.go:356] "Generic (PLEG): container finished" podID="10412ebf-7e51-446c-baa6-5c25d08383f2" containerID="98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676" exitCode=0 Jul 07 19:22:01.924685 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.924500 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" Jul 07 19:22:01.924685 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.924514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" event={"ID":"10412ebf-7e51-446c-baa6-5c25d08383f2","Type":"ContainerDied","Data":"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676"} Jul 07 19:22:01.924685 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.924558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz" event={"ID":"10412ebf-7e51-446c-baa6-5c25d08383f2","Type":"ContainerDied","Data":"db384b26ff3e73746c5b85c0eec0911638a0065ccbec9b050c729723ca59bde0"} Jul 07 19:22:01.924685 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.924577 2564 scope.go:117] "RemoveContainer" containerID="98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676" Jul 07 19:22:01.936578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.936549 2564 scope.go:117] "RemoveContainer" containerID="98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676" Jul 07 19:22:01.937129 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:01.936933 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676\": container with ID starting with 98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676 not found: ID does not exist" containerID="98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676" Jul 07 19:22:01.937129 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.936974 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676"} err="failed to get container status \"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676\": rpc error: code = NotFound desc = could not find container \"98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676\": container with ID starting with 98a027e98e5e4fdef0104572d5d11e86c51be7ecd4b98b29777971a1c794a676 not found: ID does not exist" Jul 07 19:22:01.953125 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.953083 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:22:01.959039 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:01.959007 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-2e4d95-5bdd6dcbdb-55hvz"] Jul 07 19:22:02.620561 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:02.620524 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10412ebf-7e51-446c-baa6-5c25d08383f2" path="/var/lib/kubelet/pods/10412ebf-7e51-446c-baa6-5c25d08383f2/volumes" Jul 07 19:22:03.324640 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:03.324604 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:22:04.869597 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.869568 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:22:04.941888 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.941786 2564 generic.go:356] "Generic (PLEG): container finished" podID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerID="c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad" exitCode=0 Jul 07 19:22:04.941888 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.941839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" Jul 07 19:22:04.942083 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.941876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerDied","Data":"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad"} Jul 07 19:22:04.942083 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.941937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g" event={"ID":"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667","Type":"ContainerDied","Data":"f6dae95fe9b5deb7979aaa4dec134318e0f9ec7a039167c5da28f234872c9ed4"} Jul 07 19:22:04.942083 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.941959 2564 scope.go:117] "RemoveContainer" containerID="c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad" Jul 07 19:22:04.950998 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.950978 2564 scope.go:117] "RemoveContainer" containerID="859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661" Jul 07 19:22:04.959851 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.959824 2564 scope.go:117] "RemoveContainer" containerID="c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad" Jul 07 19:22:04.960161 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:04.960129 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad\": container with ID starting with c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad not found: ID does not exist" containerID="c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad" Jul 07 19:22:04.960210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.960175 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad"} err="failed to get container status \"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad\": rpc error: code = NotFound desc = could not find container \"c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad\": container with ID starting with c13d72da0bde3f7624d8a241261880e76f56257d12cefa1884b92064b3254bad not found: ID does not exist" Jul 07 19:22:04.960210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.960196 2564 scope.go:117] "RemoveContainer" containerID="859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661" Jul 07 19:22:04.960458 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:04.960439 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661\": container with ID starting with 859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661 not found: ID does not exist" containerID="859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661" Jul 07 19:22:04.960519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:04.960468 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661"} err="failed to get container status \"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661\": rpc error: code = NotFound desc = could not find container \"859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661\": container with ID starting with 859e88ec44a33a42351bb8f1a7671c1cc1163cc44c70ec4bf9f1c74bf06cb661 not found: ID does not exist" Jul 07 19:22:05.024064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024030 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024064 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024070 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024091 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kf95\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024182 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024613 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024332 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo\") pod \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\" (UID: \"027c6f3c-4c8e-4d30-9bc2-4297c6d7e667\") " Jul 07 19:22:05.024613 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024355 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:05.024613 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024397 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:05.024613 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024567 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:05.024832 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024715 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:05.024832 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024728 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.024832 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024751 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.024832 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.024765 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.025311 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.025280 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data" (OuterVolumeSpecName: "istio-data") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:05.026740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.026716 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:22:05.026740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.026727 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:05.026895 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.026755 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95" (OuterVolumeSpecName: "kube-api-access-2kf95") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "kube-api-access-2kf95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:05.026895 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.026840 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token" (OuterVolumeSpecName: "istio-token") pod "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" (UID: "027c6f3c-4c8e-4d30-9bc2-4297c6d7e667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:05.125840 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125799 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.125840 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125836 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.125840 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125848 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kf95\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-kube-api-access-2kf95\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.126122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125860 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.126122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125872 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.126122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.125882 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:05.269062 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.269030 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:22:05.274897 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:05.274866 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0129-openshift-default-689dd64867-q577g"] Jul 07 19:22:06.621104 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:06.621068 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" path="/var/lib/kubelet/pods/027c6f3c-4c8e-4d30-9bc2-4297c6d7e667/volumes" Jul 07 19:22:07.257713 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:07.257676 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:22:07.257918 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:07.257890 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" containerID="cri-o://4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf" gracePeriod=30 Jul 07 19:22:08.164676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.164618 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:08.165052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165012 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b399460-21f8-49bf-86ba-f87fca539d6e" containerName="maas-api" Jul 07 19:22:08.165052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165024 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b399460-21f8-49bf-86ba-f87fca539d6e" containerName="maas-api" Jul 07 19:22:08.165052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165043 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10412ebf-7e51-446c-baa6-5c25d08383f2" containerName="maas-api" Jul 07 19:22:08.165052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165048 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10412ebf-7e51-446c-baa6-5c25d08383f2" containerName="maas-api" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165057 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165062 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165068 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165073 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165133 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b399460-21f8-49bf-86ba-f87fca539d6e" containerName="maas-api" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165140 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165147 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="027c6f3c-4c8e-4d30-9bc2-4297c6d7e667" containerName="istio-proxy" Jul 07 19:22:08.165210 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.165152 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="10412ebf-7e51-446c-baa6-5c25d08383f2" containerName="maas-api" Jul 07 19:22:08.168320 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.168302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.172438 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.172415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-27fc26-openshift-default-dockercfg-dnv4v\"" Jul 07 19:22:08.188082 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.188048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:08.354824 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.354824 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.354981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfdl4\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355052 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.355047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.355244 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.355077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.455940 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.455843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfdl4\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.455940 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.455915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.455961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456464 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456529 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456645 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456742 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456700 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.456742 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456730 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.457005 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.456959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.458524 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.458500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.458779 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.458758 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.466793 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.466769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfdl4\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.467424 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.467380 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token\") pod \"e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.478498 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.478464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:08.630831 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.630799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:08.632391 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:08.632358 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bf59f7_6793_4cd8_8367_5640b2ffedd7.slice/crio-4255118f92b83fd777760fe3b3638ed301eb31759a393342e4e4199d93bc46c8 WatchSource:0}: Error finding container 4255118f92b83fd777760fe3b3638ed301eb31759a393342e4e4199d93bc46c8: Status 404 returned error can't find the container with id 4255118f92b83fd777760fe3b3638ed301eb31759a393342e4e4199d93bc46c8 Jul 07 19:22:08.634754 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.634712 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:08.634844 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.634793 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:08.634844 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.634833 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:08.965177 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.965143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerStarted","Data":"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008"} Jul 07 19:22:08.965386 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.965199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerStarted","Data":"4255118f92b83fd777760fe3b3638ed301eb31759a393342e4e4199d93bc46c8"} Jul 07 19:22:08.991065 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:08.990997 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podStartSLOduration=0.99097499 podStartE2EDuration="990.97499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:08.990314192 +0000 UTC m=+1900.932936991" watchObservedRunningTime="2026-07-07 19:22:08.99097499 +0000 UTC m=+1900.933597778" Jul 07 19:22:09.479446 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:09.479404 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:09.484336 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:09.484308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:09.970262 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:09.970225 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:09.971227 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:09.971207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:11.027132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.027097 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:11.029862 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.029841 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.032758 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.032732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:22:11.032884 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.032829 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-27fc26-serving-cert\"" Jul 07 19:22:11.050415 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.050374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:11.082569 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.082532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.082808 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.082580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqv7r\" (UniqueName: \"kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.082808 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.082700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.183944 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.183898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.184131 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.183974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.184131 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.184006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqv7r\" (UniqueName: \"kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.184131 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:11.184075 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-27fc26-serving-cert: secret "maas-api-e2e-api-a-27fc26-serving-cert" not found Jul 07 19:22:11.184320 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:11.184147 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls podName:8a51f2c1-a4f7-499a-8ab1-d56806e36986 nodeName:}" failed. No retries permitted until 2026-07-07 19:22:11.684128388 +0000 UTC m=+1903.626751173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls") pod "maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" (UID: "8a51f2c1-a4f7-499a-8ab1-d56806e36986") : secret "maas-api-e2e-api-a-27fc26-serving-cert" not found Jul 07 19:22:11.184697 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.184636 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.206027 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.205986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqv7r\" (UniqueName: \"kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.688049 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.688007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.690712 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.690681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") pod \"maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:11.941306 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:11.941207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:12.095335 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.095236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:12.097602 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:12.097570 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a51f2c1_a4f7_499a_8ab1_d56806e36986.slice/crio-0e76601e3bb64cb302c56eb097bb5b8893d5cc0cd5e7ea9ce8b80d4b43986602 WatchSource:0}: Error finding container 0e76601e3bb64cb302c56eb097bb5b8893d5cc0cd5e7ea9ce8b80d4b43986602: Status 404 returned error can't find the container with id 0e76601e3bb64cb302c56eb097bb5b8893d5cc0cd5e7ea9ce8b80d4b43986602 Jul 07 19:22:12.519548 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.519473 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:22:12.598831 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598782 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.598831 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598857 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcxrk\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598921 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599071 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.598949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:12.599316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert\") pod \"f5b80cc2-ff20-4451-8de4-672e549fa948\" (UID: \"f5b80cc2-ff20-4451-8de4-672e549fa948\") " Jul 07 19:22:12.599316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599261 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:12.599506 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599396 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:12.599680 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599633 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:12.599680 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599638 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.599680 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599681 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.599980 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.599696 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.600048 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.600031 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:12.601963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.601933 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:22:12.602073 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.601992 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:12.602073 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.602037 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:12.602149 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.602092 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk" (OuterVolumeSpecName: "kube-api-access-lcxrk") pod "f5b80cc2-ff20-4451-8de4-672e549fa948" (UID: "f5b80cc2-ff20-4451-8de4-672e549fa948"). InnerVolumeSpecName "kube-api-access-lcxrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:12.701712 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701631 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b80cc2-ff20-4451-8de4-672e549fa948-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.701712 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701700 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.701712 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701715 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.701918 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701731 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxrk\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-kube-api-access-lcxrk\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.701918 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701749 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.701918 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.701763 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b80cc2-ff20-4451-8de4-672e549fa948-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:12.987507 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.987467 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerID="4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf" exitCode=0 Jul 07 19:22:12.987741 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.987524 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" Jul 07 19:22:12.987741 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.987548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerDied","Data":"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf"} Jul 07 19:22:12.987741 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.987611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h" event={"ID":"f5b80cc2-ff20-4451-8de4-672e549fa948","Type":"ContainerDied","Data":"7d985038ff9c6dc475800d9ca1e9d1e0c5223e8eaba6cd59ab0e7a82483e34f9"} Jul 07 19:22:12.987741 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.987629 2564 scope.go:117] "RemoveContainer" containerID="4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf" Jul 07 19:22:12.989611 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.989583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" event={"ID":"8a51f2c1-a4f7-499a-8ab1-d56806e36986","Type":"ContainerStarted","Data":"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d"} Jul 07 19:22:12.989757 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.989638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" event={"ID":"8a51f2c1-a4f7-499a-8ab1-d56806e36986","Type":"ContainerStarted","Data":"0e76601e3bb64cb302c56eb097bb5b8893d5cc0cd5e7ea9ce8b80d4b43986602"} Jul 07 19:22:12.989889 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.989852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:12.996690 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:12.996633 2564 scope.go:117] "RemoveContainer" containerID="b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621" Jul 07 19:22:13.005295 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.005268 2564 scope.go:117] "RemoveContainer" containerID="4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf" Jul 07 19:22:13.005629 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:13.005606 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf\": container with ID starting with 4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf not found: ID does not exist" containerID="4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf" Jul 07 19:22:13.005736 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.005637 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf"} err="failed to get container status \"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf\": rpc error: code = NotFound desc = could not find container \"4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf\": container with ID starting with 4e1f1d2c71a1292c3377efc3f567b38d2368ddb7e9b59ed55a3b6c331b4d5acf not found: ID does not exist" Jul 07 19:22:13.005736 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.005686 2564 scope.go:117] "RemoveContainer" containerID="b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621" Jul 07 19:22:13.005949 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:13.005934 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621\": container with ID starting with b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621 not found: ID does not exist" containerID="b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621" Jul 07 19:22:13.005991 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.005953 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621"} err="failed to get container status \"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621\": rpc error: code = NotFound desc = could not find container \"b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621\": container with ID starting with b6baa99caf362d0c95ac1868b3a488473afbcf01ef655d52bf7781616c535621 not found: ID does not exist" Jul 07 19:22:13.012362 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.012313 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" podStartSLOduration=2.47737428 podStartE2EDuration="3.012299048s" totalImagesPullingTime="534.924768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:13.011677203 +0000 UTC m=+1904.954299990" watchObservedRunningTime="2026-07-07 19:22:13.012299048 +0000 UTC m=+1904.954921834" Jul 07 19:22:13.034688 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.034631 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:22:13.039534 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:13.039506 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4d95-openshift-default-854775cdf-dgj7h"] Jul 07 19:22:14.620462 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:14.620428 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" path="/var/lib/kubelet/pods/f5b80cc2-ff20-4451-8de4-672e549fa948/volumes" Jul 07 19:22:15.840905 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:15.840870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:22:16.247637 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.247549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:16.247995 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.247981 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:16.248072 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.247998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:16.248134 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.248076 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:16.248134 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.248090 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:16.251177 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.251151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.253701 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.253674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c129c9-openshift-default-dockercfg-4gdzc\"" Jul 07 19:22:16.270128 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.270098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:16.340552 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340765 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340765 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340765 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66wr\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.340948 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.340931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442248 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442248 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66wr\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442523 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442949 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442800 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.442949 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.443013 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.442993 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.443048 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.443013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.443312 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.443294 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.445455 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.445423 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.445552 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.445538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.460089 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.460052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.461127 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.461107 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66wr\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr\") pod \"e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.563983 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.563950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:16.722677 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.722634 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:16.725322 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:16.725289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eae245e_ff89_4477_a7f9_a6d2d1fb72b8.slice/crio-83fd9f4c365792a675be1dabcdc9cd0a277f78fb9592403575c005ab1e6ca505 WatchSource:0}: Error finding container 83fd9f4c365792a675be1dabcdc9cd0a277f78fb9592403575c005ab1e6ca505: Status 404 returned error can't find the container with id 83fd9f4c365792a675be1dabcdc9cd0a277f78fb9592403575c005ab1e6ca505 Jul 07 19:22:16.727699 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.727640 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:16.727795 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.727735 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:16.727795 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:16.727766 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:17.013454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:17.013340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerStarted","Data":"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e"} Jul 07 19:22:17.013454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:17.013411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerStarted","Data":"83fd9f4c365792a675be1dabcdc9cd0a277f78fb9592403575c005ab1e6ca505"} Jul 07 19:22:17.050269 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:17.050174 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podStartSLOduration=1.050151181 podStartE2EDuration="1.050151181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:17.050150831 +0000 UTC m=+1908.992773621" watchObservedRunningTime="2026-07-07 19:22:17.050151181 +0000 UTC m=+1908.992773968" Jul 07 19:22:17.565106 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:17.565061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:17.570311 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:17.570282 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:18.022788 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:18.022643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:18.027628 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:18.027590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:19.001957 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.001929 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:19.271098 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.271017 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:19.271453 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.271405 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:19.271453 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.271416 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b80cc2-ff20-4451-8de4-672e549fa948" containerName="istio-proxy" Jul 07 19:22:19.273805 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.273786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.277886 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.277862 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-c129c9-serving-cert\"" Jul 07 19:22:19.301813 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.301775 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:19.372198 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.372151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzhw\" (UniqueName: \"kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.372404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.372213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.372404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.372255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.473426 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.473359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzhw\" (UniqueName: \"kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.473616 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.473486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.473616 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.473545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.475041 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.475008 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.476688 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.476630 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.506545 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.506486 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzhw\" (UniqueName: \"kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw\") pod \"maas-api-e2e-api-b-c129c9-8666846559-7kqzb\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.584783 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.584746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:19.804859 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:19.804803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:19.807318 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:19.807201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290d0cc8_fcb7_4c82_ab63_091b131760b0.slice/crio-7d75607d0bc8bc9897bc40d8c534a45d0efe685b3e00f97c26cd8a2cda7a35cd WatchSource:0}: Error finding container 7d75607d0bc8bc9897bc40d8c534a45d0efe685b3e00f97c26cd8a2cda7a35cd: Status 404 returned error can't find the container with id 7d75607d0bc8bc9897bc40d8c534a45d0efe685b3e00f97c26cd8a2cda7a35cd Jul 07 19:22:20.040300 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:20.040261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" event={"ID":"290d0cc8-fcb7-4c82-ab63-091b131760b0","Type":"ContainerStarted","Data":"7d75607d0bc8bc9897bc40d8c534a45d0efe685b3e00f97c26cd8a2cda7a35cd"} Jul 07 19:22:21.049207 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:21.049173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" event={"ID":"290d0cc8-fcb7-4c82-ab63-091b131760b0","Type":"ContainerStarted","Data":"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af"} Jul 07 19:22:21.049728 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:21.049284 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:21.069802 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:21.069735 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" podStartSLOduration=1.472594421 podStartE2EDuration="2.069712598s" totalImagesPullingTime="597.118177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:21.068303288 +0000 UTC m=+1913.010926077" watchObservedRunningTime="2026-07-07 19:22:21.069712598 +0000 UTC m=+1913.012335387" Jul 07 19:22:24.069128 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.069096 2564 generic.go:356] "Generic (PLEG): container finished" podID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerID="5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008" exitCode=0 Jul 07 19:22:24.069590 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.069157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerDied","Data":"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008"} Jul 07 19:22:24.069590 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.069566 2564 scope.go:117] "RemoveContainer" containerID="5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008" Jul 07 19:22:24.070676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.070613 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:24.070777 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.070720 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:24.070777 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.070756 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:24.525310 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.525267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:22:24.971302 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:24.971253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:25.076374 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:25.076331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerStarted","Data":"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005"} Jul 07 19:22:25.479695 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:25.479618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:25.480943 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:25.480916 2564 patch_prober.go:28] interesting pod/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:22:25.481023 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:25.480972 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 07 19:22:26.479553 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:26.479513 2564 patch_prober.go:28] interesting pod/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:22:26.479991 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:26.479569 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 07 19:22:27.067448 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:27.067414 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:27.479447 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:27.479350 2564 patch_prober.go:28] interesting pod/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:22:27.479447 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:27.479420 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 07 19:22:28.479834 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:28.479786 2564 patch_prober.go:28] interesting pod/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:22:28.480229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:28.479867 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 07 19:22:29.484131 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:29.484093 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:29.484562 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:29.484478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:29.485117 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:29.485099 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:32.336675 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.331775 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:32.336675 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.332066 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" podUID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" containerName="maas-api" containerID="cri-o://ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d" gracePeriod=30 Jul 07 19:22:32.634351 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.634325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:32.725127 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.725033 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca\") pod \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " Jul 07 19:22:32.725127 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.725084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") pod \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " Jul 07 19:22:32.725397 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.725174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqv7r\" (UniqueName: \"kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r\") pod \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\" (UID: \"8a51f2c1-a4f7-499a-8ab1-d56806e36986\") " Jul 07 19:22:32.725461 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.725418 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a51f2c1-a4f7-499a-8ab1-d56806e36986" (UID: "8a51f2c1-a4f7-499a-8ab1-d56806e36986"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:32.725548 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.725524 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a51f2c1-a4f7-499a-8ab1-d56806e36986-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:32.727533 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.727498 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a51f2c1-a4f7-499a-8ab1-d56806e36986" (UID: "8a51f2c1-a4f7-499a-8ab1-d56806e36986"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:22:32.727634 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.727553 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r" (OuterVolumeSpecName: "kube-api-access-zqv7r") pod "8a51f2c1-a4f7-499a-8ab1-d56806e36986" (UID: "8a51f2c1-a4f7-499a-8ab1-d56806e36986"). InnerVolumeSpecName "kube-api-access-zqv7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:32.826319 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.826278 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a51f2c1-a4f7-499a-8ab1-d56806e36986-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:32.826319 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:32.826312 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqv7r\" (UniqueName: \"kubernetes.io/projected/8a51f2c1-a4f7-499a-8ab1-d56806e36986-kube-api-access-zqv7r\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:33.024126 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.024034 2564 patch_prober.go:28] interesting pod/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:22:33.024126 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.024106 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 19:22:33.120230 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.120190 2564 generic.go:356] "Generic (PLEG): container finished" podID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" containerID="ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d" exitCode=0 Jul 07 19:22:33.120410 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.120244 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" Jul 07 19:22:33.120410 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.120275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" event={"ID":"8a51f2c1-a4f7-499a-8ab1-d56806e36986","Type":"ContainerDied","Data":"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d"} Jul 07 19:22:33.120410 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.120337 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh" event={"ID":"8a51f2c1-a4f7-499a-8ab1-d56806e36986","Type":"ContainerDied","Data":"0e76601e3bb64cb302c56eb097bb5b8893d5cc0cd5e7ea9ce8b80d4b43986602"} Jul 07 19:22:33.120410 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.120362 2564 scope.go:117] "RemoveContainer" containerID="ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d" Jul 07 19:22:33.122054 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.122028 2564 generic.go:356] "Generic (PLEG): container finished" podID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerID="32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e" exitCode=0 Jul 07 19:22:33.122185 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.122060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerDied","Data":"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e"} Jul 07 19:22:33.122455 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.122441 2564 scope.go:117] "RemoveContainer" containerID="32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e" Jul 07 19:22:33.123643 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.123606 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:33.123756 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.123740 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:33.123816 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.123791 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:33.129966 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.129945 2564 scope.go:117] "RemoveContainer" containerID="ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d" Jul 07 19:22:33.130257 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:33.130238 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d\": container with ID starting with ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d not found: ID does not exist" containerID="ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d" Jul 07 19:22:33.130316 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.130266 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d"} err="failed to get container status \"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d\": rpc error: code = NotFound desc = could not find container \"ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d\": container with ID starting with ebdfba88d3021ec6b3955cfd49fd3b63b35342802c930a32f71d8c0f0416197d not found: ID does not exist" Jul 07 19:22:33.165974 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.165935 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:33.169979 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:33.169947 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-27fc26-57dd7986cc-pbmwh"] Jul 07 19:22:34.135644 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.135581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerStarted","Data":"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e"} Jul 07 19:22:34.564234 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.564191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:34.564458 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.564435 2564 patch_prober.go:28] interesting pod/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:22:34.564514 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.564486 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 19:22:34.625222 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.625174 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" path="/var/lib/kubelet/pods/8a51f2c1-a4f7-499a-8ab1-d56806e36986/volumes" Jul 07 19:22:34.853243 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:34.853155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:22:35.565073 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:35.565036 2564 patch_prober.go:28] interesting pod/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:22:35.565498 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:35.565093 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 19:22:36.565235 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:36.565195 2564 patch_prober.go:28] interesting pod/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:22:36.565749 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:36.565278 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 19:22:37.564557 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:37.564516 2564 patch_prober.go:28] interesting pod/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:22:37.564780 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:37.564582 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 07 19:22:38.507667 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:38.507613 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:38.508109 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:38.507888 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" containerID="cri-o://6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005" gracePeriod=30 Jul 07 19:22:38.569151 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:38.569120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:38.569635 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:38.569606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:38.570286 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:38.570269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:39.706390 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:39.706357 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:39.706825 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:39.706571 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" podUID="290d0cc8-fcb7-4c82-ab63-091b131760b0" containerName="maas-api" containerID="cri-o://093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af" gracePeriod=30 Jul 07 19:22:39.953842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:39.953815 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:40.099031 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.098992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca\") pod \"290d0cc8-fcb7-4c82-ab63-091b131760b0\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " Jul 07 19:22:40.099221 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.099091 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqzhw\" (UniqueName: \"kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw\") pod \"290d0cc8-fcb7-4c82-ab63-091b131760b0\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " Jul 07 19:22:40.099221 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.099121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls\") pod \"290d0cc8-fcb7-4c82-ab63-091b131760b0\" (UID: \"290d0cc8-fcb7-4c82-ab63-091b131760b0\") " Jul 07 19:22:40.099456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.099427 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "290d0cc8-fcb7-4c82-ab63-091b131760b0" (UID: "290d0cc8-fcb7-4c82-ab63-091b131760b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:40.101317 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.101289 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw" (OuterVolumeSpecName: "kube-api-access-kqzhw") pod "290d0cc8-fcb7-4c82-ab63-091b131760b0" (UID: "290d0cc8-fcb7-4c82-ab63-091b131760b0"). InnerVolumeSpecName "kube-api-access-kqzhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:40.101460 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.101409 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "290d0cc8-fcb7-4c82-ab63-091b131760b0" (UID: "290d0cc8-fcb7-4c82-ab63-091b131760b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:22:40.168149 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.168112 2564 generic.go:356] "Generic (PLEG): container finished" podID="290d0cc8-fcb7-4c82-ab63-091b131760b0" containerID="093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af" exitCode=0 Jul 07 19:22:40.168353 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.168161 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" Jul 07 19:22:40.168353 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.168206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" event={"ID":"290d0cc8-fcb7-4c82-ab63-091b131760b0","Type":"ContainerDied","Data":"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af"} Jul 07 19:22:40.168353 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.168258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb" event={"ID":"290d0cc8-fcb7-4c82-ab63-091b131760b0","Type":"ContainerDied","Data":"7d75607d0bc8bc9897bc40d8c534a45d0efe685b3e00f97c26cd8a2cda7a35cd"} Jul 07 19:22:40.168353 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.168288 2564 scope.go:117] "RemoveContainer" containerID="093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af" Jul 07 19:22:40.177462 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.177444 2564 scope.go:117] "RemoveContainer" containerID="093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af" Jul 07 19:22:40.177788 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:40.177766 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af\": container with ID starting with 093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af not found: ID does not exist" containerID="093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af" Jul 07 19:22:40.177859 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.177797 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af"} err="failed to get container status \"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af\": rpc error: code = NotFound desc = could not find container \"093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af\": container with ID starting with 093187b2ce026e93baed1eb0c27f7d9b8fc8e5143d43213740a21d156b54a7af not found: ID does not exist" Jul 07 19:22:40.191003 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.190972 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:40.196154 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.196130 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c129c9-8666846559-7kqzb"] Jul 07 19:22:40.200178 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.200156 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzhw\" (UniqueName: \"kubernetes.io/projected/290d0cc8-fcb7-4c82-ab63-091b131760b0-kube-api-access-kqzhw\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:40.200289 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.200183 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290d0cc8-fcb7-4c82-ab63-091b131760b0-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:40.200289 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.200194 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290d0cc8-fcb7-4c82-ab63-091b131760b0-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:40.620744 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:40.620710 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290d0cc8-fcb7-4c82-ab63-091b131760b0" path="/var/lib/kubelet/pods/290d0cc8-fcb7-4c82-ab63-091b131760b0/volumes" Jul 07 19:22:43.768089 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.768018 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:43.829620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829574 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829903 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829903 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfdl4\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829903 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829804 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829903 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.829903 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.830160 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829898 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:43.830160 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829909 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.830160 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.829995 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo\") pod \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\" (UID: \"62bf59f7-6793-4cd8-8367-5640b2ffedd7\") " Jul 07 19:22:43.830305 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830128 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:43.830305 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830212 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:43.830404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830359 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.830404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830381 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.830404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830394 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.830555 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830449 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:43.830827 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.830781 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data" (OuterVolumeSpecName: "istio-data") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:43.832274 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.832247 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:43.832377 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.832358 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:22:43.832621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.832595 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token" (OuterVolumeSpecName: "istio-token") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:43.832621 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.832611 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4" (OuterVolumeSpecName: "kube-api-access-wfdl4") pod "62bf59f7-6793-4cd8-8367-5640b2ffedd7" (UID: "62bf59f7-6793-4cd8-8367-5640b2ffedd7"). InnerVolumeSpecName "kube-api-access-wfdl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:43.930975 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.930932 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.930975 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.930968 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfdl4\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-kube-api-access-wfdl4\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.930975 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.930979 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.930975 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.930987 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.931261 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.930995 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:43.931261 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:43.931002 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bf59f7-6793-4cd8-8367-5640b2ffedd7-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:44.191487 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.191449 2564 generic.go:356] "Generic (PLEG): container finished" podID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerID="6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005" exitCode=0 Jul 07 19:22:44.191704 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.191497 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" Jul 07 19:22:44.191704 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.191521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerDied","Data":"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005"} Jul 07 19:22:44.191704 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.191570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw" event={"ID":"62bf59f7-6793-4cd8-8367-5640b2ffedd7","Type":"ContainerDied","Data":"4255118f92b83fd777760fe3b3638ed301eb31759a393342e4e4199d93bc46c8"} Jul 07 19:22:44.191704 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.191594 2564 scope.go:117] "RemoveContainer" containerID="6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005" Jul 07 19:22:44.201057 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.201038 2564 scope.go:117] "RemoveContainer" containerID="5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008" Jul 07 19:22:44.209173 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.209149 2564 scope.go:117] "RemoveContainer" containerID="6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005" Jul 07 19:22:44.209493 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:44.209473 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005\": container with ID starting with 6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005 not found: ID does not exist" containerID="6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005" Jul 07 19:22:44.209575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.209506 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005"} err="failed to get container status \"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005\": rpc error: code = NotFound desc = could not find container \"6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005\": container with ID starting with 6dd9be0435d1fadedffbb5620ce1364117630f7d7dcacc10befcdfab899fa005 not found: ID does not exist" Jul 07 19:22:44.209575 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.209533 2564 scope.go:117] "RemoveContainer" containerID="5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008" Jul 07 19:22:44.209816 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:44.209798 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008\": container with ID starting with 5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008 not found: ID does not exist" containerID="5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008" Jul 07 19:22:44.209861 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.209822 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008"} err="failed to get container status \"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008\": rpc error: code = NotFound desc = could not find container \"5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008\": container with ID starting with 5a4e2d46120d709ee972230f8975b6c872d78bfe7ba00bd871d1165183293008 not found: ID does not exist" Jul 07 19:22:44.216334 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.216310 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:44.222180 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.222155 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-27fc26-openshift-default-6b8ff68cb7-gtdgw"] Jul 07 19:22:44.620366 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:44.620320 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" path="/var/lib/kubelet/pods/62bf59f7-6793-4cd8-8367-5640b2ffedd7/volumes" Jul 07 19:22:46.178214 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:46.178171 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:46.178678 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:46.178432 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" containerID="cri-o://8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e" gracePeriod=30 Jul 07 19:22:47.054882 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.054842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:22:47.055243 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055230 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" containerName="maas-api" Jul 07 19:22:47.055283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055246 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" containerName="maas-api" Jul 07 19:22:47.055283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055253 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.055283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055259 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.055283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055268 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.055283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055275 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055286 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290d0cc8-fcb7-4c82-ab63-091b131760b0" containerName="maas-api" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055291 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="290d0cc8-fcb7-4c82-ab63-091b131760b0" containerName="maas-api" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055348 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="290d0cc8-fcb7-4c82-ab63-091b131760b0" containerName="maas-api" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055359 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a51f2c1-a4f7-499a-8ab1-d56806e36986" containerName="maas-api" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055367 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.055451 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.055372 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="62bf59f7-6793-4cd8-8367-5640b2ffedd7" containerName="istio-proxy" Jul 07 19:22:47.058219 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.058196 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.061391 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.061325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6e21d5-openshift-default-dockercfg-4phmj\"" Jul 07 19:22:47.084392 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.084359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:22:47.161194 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161194 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhml7\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161456 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.161681 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.161474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhml7\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.262891 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262836 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.263298 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.262966 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.263298 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.263006 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.263298 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.263137 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.263298 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.263204 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.265222 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.265198 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.265401 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.265366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.273530 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.273502 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.273676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.273621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhml7\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7\") pod \"e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.368996 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.368893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:47.523385 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.523356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:22:47.525295 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:47.525269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53d9df3_888c_4af1_b025_5635c5a1865a.slice/crio-642694aceee5beade47f088e570429f97aa9f149e6917c1010fca0de99c5e84a WatchSource:0}: Error finding container 642694aceee5beade47f088e570429f97aa9f149e6917c1010fca0de99c5e84a: Status 404 returned error can't find the container with id 642694aceee5beade47f088e570429f97aa9f149e6917c1010fca0de99c5e84a Jul 07 19:22:47.527540 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.527502 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:47.527631 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.527589 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:47.527707 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:47.527635 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:48.214643 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.214602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerStarted","Data":"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2"} Jul 07 19:22:48.214879 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.214681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerStarted","Data":"642694aceee5beade47f088e570429f97aa9f149e6917c1010fca0de99c5e84a"} Jul 07 19:22:48.239842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.239788 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podStartSLOduration=1.239770612 podStartE2EDuration="1.239770612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:48.238021919 +0000 UTC m=+1940.180644708" watchObservedRunningTime="2026-07-07 19:22:48.239770612 +0000 UTC m=+1940.182393399" Jul 07 19:22:48.369709 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.369644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:48.374624 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.374593 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:48.733624 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:48.733587 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:22:49.219964 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:49.219931 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:49.220912 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:49.220892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:22:50.235811 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.235778 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:22:50.238525 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.238502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.242262 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.242244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxp29\"" Jul 07 19:22:50.242423 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.242395 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-6e21d5-serving-cert\"" Jul 07 19:22:50.253242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.253209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:22:50.295694 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.295612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.295860 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.295803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9k6b\" (UniqueName: \"kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.295940 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.295855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.396929 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.396890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.397134 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.396970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9k6b\" (UniqueName: \"kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.397134 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.396999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.397777 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.397755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.399546 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.399521 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.411984 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.411952 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9k6b\" (UniqueName: \"kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b\") pod \"maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.549958 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.549899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:50.698400 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:50.698373 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:22:50.700509 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:50.700483 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2e0b7b_e4cf_4a22_a758_822c94ffa4eb.slice/crio-071e2d5f196d13ede3af25fac02f312bc7af63fde3ab645611b770212baf1bc3 WatchSource:0}: Error finding container 071e2d5f196d13ede3af25fac02f312bc7af63fde3ab645611b770212baf1bc3: Status 404 returned error can't find the container with id 071e2d5f196d13ede3af25fac02f312bc7af63fde3ab645611b770212baf1bc3 Jul 07 19:22:51.233542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.233502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" event={"ID":"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb","Type":"ContainerStarted","Data":"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb"} Jul 07 19:22:51.233743 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.233604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" event={"ID":"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb","Type":"ContainerStarted","Data":"071e2d5f196d13ede3af25fac02f312bc7af63fde3ab645611b770212baf1bc3"} Jul 07 19:22:51.233808 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.233794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:51.274065 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.273986 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" podStartSLOduration=0.859047934 podStartE2EDuration="1.273970155s" totalImagesPullingTime="414.922221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:51.272439123 +0000 UTC m=+1943.215061910" watchObservedRunningTime="2026-07-07 19:22:51.273970155 +0000 UTC m=+1943.216592943" Jul 07 19:22:51.442459 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.442432 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:51.508216 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508180 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508216 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508225 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508242 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508324 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z66wr\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508454 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508469 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs\") pod \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\" (UID: \"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8\") " Jul 07 19:22:51.508798 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508559 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:51.508902 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508824 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:51.508955 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508903 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.508955 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.508926 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.509137 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.509103 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:51.509424 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.509143 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:51.509424 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.509233 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:22:51.510928 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.510903 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr" (OuterVolumeSpecName: "kube-api-access-z66wr") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "kube-api-access-z66wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:51.510928 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.510909 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:22:51.511167 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.511144 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:22:51.511167 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.511155 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" (UID: "5eae245e-ff89-4477-a7f9-a6d2d1fb72b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:51.609842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609804 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.609842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609835 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.609842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609844 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.609842 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609852 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66wr\" (UniqueName: \"kubernetes.io/projected/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-kube-api-access-z66wr\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.610115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609863 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.610115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609872 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:51.610115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:51.609880 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:22:52.240616 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.240581 2564 generic.go:356] "Generic (PLEG): container finished" podID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerID="8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e" exitCode=0 Jul 07 19:22:52.240853 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.240634 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" Jul 07 19:22:52.240853 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.240692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerDied","Data":"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e"} Jul 07 19:22:52.240853 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.240757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg" event={"ID":"5eae245e-ff89-4477-a7f9-a6d2d1fb72b8","Type":"ContainerDied","Data":"83fd9f4c365792a675be1dabcdc9cd0a277f78fb9592403575c005ab1e6ca505"} Jul 07 19:22:52.240853 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.240788 2564 scope.go:117] "RemoveContainer" containerID="8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e" Jul 07 19:22:52.250539 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.250517 2564 scope.go:117] "RemoveContainer" containerID="32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e" Jul 07 19:22:52.259187 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.259163 2564 scope.go:117] "RemoveContainer" containerID="8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e" Jul 07 19:22:52.259507 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:52.259487 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e\": container with ID starting with 8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e not found: ID does not exist" containerID="8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e" Jul 07 19:22:52.259566 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.259517 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e"} err="failed to get container status \"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e\": rpc error: code = NotFound desc = could not find container \"8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e\": container with ID starting with 8788143c5ac28fdac5299340fc0a9ab63a981f9f1ffa8c6d75c0a78b34cca15e not found: ID does not exist" Jul 07 19:22:52.259566 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.259540 2564 scope.go:117] "RemoveContainer" containerID="32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e" Jul 07 19:22:52.259860 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:22:52.259840 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e\": container with ID starting with 32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e not found: ID does not exist" containerID="32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e" Jul 07 19:22:52.259967 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.259864 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e"} err="failed to get container status \"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e\": rpc error: code = NotFound desc = could not find container \"32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e\": container with ID starting with 32a96d6628740592cf415a41943ecce4a477140fe2808cf0870f1dc7f620b01e not found: ID does not exist" Jul 07 19:22:52.263379 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.263351 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:52.267383 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.267356 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c129c9-openshift-default-6d9f465c79-bhxwg"] Jul 07 19:22:52.620824 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:52.620792 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" path="/var/lib/kubelet/pods/5eae245e-ff89-4477-a7f9-a6d2d1fb72b8/volumes" Jul 07 19:22:55.247546 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.247506 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:22:55.248136 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248112 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.248136 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248136 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.248320 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248157 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.248320 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248166 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.248320 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248264 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.248320 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.248277 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eae245e-ff89-4477-a7f9-a6d2d1fb72b8" containerName="istio-proxy" Jul 07 19:22:55.251863 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.251839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.254096 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.254076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-912fa0-openshift-default-dockercfg-pjd5j\"" Jul 07 19:22:55.263186 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.263157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.348759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.348618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzlg\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449297 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449297 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449589 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzlg\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449978 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449895 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.449978 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.449929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.450091 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.450065 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.450140 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.450112 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.450724 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.450701 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.451936 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.451905 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.452032 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.452010 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.458855 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.458823 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.458978 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.458959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzlg\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg\") pod \"e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.567165 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.567127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:55.738769 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.737324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:22:55.743872 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:55.743839 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8b667b_4184_4326_b810_4ef6b4d56e04.slice/crio-06e8ccde8ba003c24f7921cb792a7beaf7650bd818b066028085c21650c2af09 WatchSource:0}: Error finding container 06e8ccde8ba003c24f7921cb792a7beaf7650bd818b066028085c21650c2af09: Status 404 returned error can't find the container with id 06e8ccde8ba003c24f7921cb792a7beaf7650bd818b066028085c21650c2af09 Jul 07 19:22:55.746325 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.746291 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:55.746408 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.746364 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:55.746408 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:55.746394 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:22:56.269630 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:56.269591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerStarted","Data":"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83"} Jul 07 19:22:56.270069 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:56.269693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerStarted","Data":"06e8ccde8ba003c24f7921cb792a7beaf7650bd818b066028085c21650c2af09"} Jul 07 19:22:56.294365 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:56.294308 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podStartSLOduration=1.294291079 podStartE2EDuration="1.294291079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:56.292220758 +0000 UTC m=+1948.234843546" watchObservedRunningTime="2026-07-07 19:22:56.294291079 +0000 UTC m=+1948.236913867" Jul 07 19:22:56.567461 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:56.567425 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:56.572542 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:56.572510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:57.247509 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:57.247477 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:22:57.274938 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:57.274906 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:57.275907 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:57.275884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:22:58.458509 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.458472 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:22:58.461061 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.461043 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.466244 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.466219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-912fa0-serving-cert\"" Jul 07 19:22:58.475906 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.475870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:22:58.582519 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.582482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.582763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.582540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.582763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.582729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w42h\" (UniqueName: \"kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.684198 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.684157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w42h\" (UniqueName: \"kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.684361 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.684247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.684733 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.684435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.685110 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.685088 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.686971 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.686945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.713739 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.713632 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w42h\" (UniqueName: \"kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h\") pod \"maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.771838 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.771799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:22:58.965411 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:58.965360 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:22:58.967267 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:22:58.967241 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a474be1_e77c_4667_bf4e_7f909d932628.slice/crio-8c2a57a59583c7bad2ca310a4b0cbe59c7a5899a970d62a9bffd4d90acd626f1 WatchSource:0}: Error finding container 8c2a57a59583c7bad2ca310a4b0cbe59c7a5899a970d62a9bffd4d90acd626f1: Status 404 returned error can't find the container with id 8c2a57a59583c7bad2ca310a4b0cbe59c7a5899a970d62a9bffd4d90acd626f1 Jul 07 19:22:59.289553 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:59.289520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" event={"ID":"4a474be1-e77c-4667-bf4e-7f909d932628","Type":"ContainerStarted","Data":"8c2a57a59583c7bad2ca310a4b0cbe59c7a5899a970d62a9bffd4d90acd626f1"} Jul 07 19:22:59.424774 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:22:59.424728 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:23:00.296794 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:00.296749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" event={"ID":"4a474be1-e77c-4667-bf4e-7f909d932628","Type":"ContainerStarted","Data":"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc"} Jul 07 19:23:00.297215 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:00.296833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:23:00.317585 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:00.317525 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" podStartSLOduration=2.013017788 podStartE2EDuration="2.317498734s" totalImagesPullingTime="304.480946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:23:00.315256411 +0000 UTC m=+1952.257879198" watchObservedRunningTime="2026-07-07 19:23:00.317498734 +0000 UTC m=+1952.260121585" Jul 07 19:23:03.313149 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.313111 2564 generic.go:356] "Generic (PLEG): container finished" podID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerID="dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2" exitCode=0 Jul 07 19:23:03.313676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.313184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerDied","Data":"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2"} Jul 07 19:23:03.313676 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.313669 2564 scope.go:117] "RemoveContainer" containerID="dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2" Jul 07 19:23:03.314738 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.314706 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:03.314852 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.314792 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:03.314852 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:03.314842 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:04.220190 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:04.220135 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:23:04.324756 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:04.324707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerStarted","Data":"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd"} Jul 07 19:23:04.369276 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:04.369236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:23:04.370382 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:04.370356 2564 patch_prober.go:28] interesting pod/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:23:04.370492 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:04.370407 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:23:05.370109 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:05.370066 2564 patch_prober.go:28] interesting pod/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:23:05.370541 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:05.370133 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:23:06.307892 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:06.307860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:23:06.373231 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:06.373189 2564 patch_prober.go:28] interesting pod/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:23:06.373901 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:06.373863 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:23:07.370107 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:07.370066 2564 patch_prober.go:28] interesting pod/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:23:07.370297 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:07.370120 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:23:08.370040 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:08.370000 2564 patch_prober.go:28] interesting pod/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:23:08.370452 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:08.370066 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:23:09.374276 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:09.374249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:23:09.374693 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:09.374586 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:23:09.375464 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:09.375443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:23:09.627263 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:09.627141 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:23:12.904578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:12.903826 2564 patch_prober.go:28] interesting pod/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:46702->10.134.0.51:15021: read: connection reset by peer" start-of-body= Jul 07 19:23:12.904578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:12.903892 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:46702->10.134.0.51:15021: read: connection reset by peer" Jul 07 19:23:13.399343 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.399302 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerID="7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83" exitCode=0 Jul 07 19:23:13.399564 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.399373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerDied","Data":"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83"} Jul 07 19:23:13.399868 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.399855 2564 scope.go:117] "RemoveContainer" containerID="7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83" Jul 07 19:23:13.400922 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.400893 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:13.400993 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.400972 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:13.401045 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:13.401003 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:23:14.405980 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:14.405943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerStarted","Data":"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b"} Jul 07 19:23:14.568115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:14.568079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:23:14.568329 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:14.568305 2564 patch_prober.go:28] interesting pod/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:23:14.568372 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:14.568357 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:23:15.568552 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:15.568513 2564 patch_prober.go:28] interesting pod/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:23:15.568983 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:15.568577 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:23:16.568300 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:16.568262 2564 patch_prober.go:28] interesting pod/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:23:16.568511 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:16.568326 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:23:17.568140 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:17.568093 2564 patch_prober.go:28] interesting pod/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:23:17.568563 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:17.568158 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:23:18.572238 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:18.572156 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:23:18.572639 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:18.572448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:23:18.573132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:18.573113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:23:18.916795 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:18.916694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:23:33.512406 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:33.512361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:23:48.433719 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:23:48.433675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:24:01.124751 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:24:01.124705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:24:17.922017 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:24:17.921977 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:24:34.324237 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:24:34.324190 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:24:49.122191 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:24:49.122099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:25:02.721725 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:25:02.721683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:25:16.432714 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:25:16.432678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:25:29.929037 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:25:29.928997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:25:45.723915 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:25:45.723873 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:26:15.266881 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:26:15.266840 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:26:33.017463 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:26:33.017382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:26:51.025813 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:26:51.025772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:27:10.325790 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:27:10.325748 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:27:39.227919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:27:39.227881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:27:58.931730 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:27:58.931631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:28:12.924020 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:12.923894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:28:21.914932 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:21.914889 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:28:21.915391 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:21.915105 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" podUID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" containerName="maas-api" containerID="cri-o://fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb" gracePeriod=30 Jul 07 19:28:22.169865 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.169789 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:28:22.189027 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.188988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9k6b\" (UniqueName: \"kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b\") pod \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " Jul 07 19:28:22.189215 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.189056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca\") pod \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " Jul 07 19:28:22.189215 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.189127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls\") pod \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\" (UID: \"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb\") " Jul 07 19:28:22.189875 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.189786 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" (UID: "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:28:22.191805 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.191757 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b" (OuterVolumeSpecName: "kube-api-access-s9k6b") pod "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" (UID: "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb"). InnerVolumeSpecName "kube-api-access-s9k6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:28:22.192059 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.192034 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" (UID: "ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:28:22.290632 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.290590 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9k6b\" (UniqueName: \"kubernetes.io/projected/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-kube-api-access-s9k6b\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:28:22.290632 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.290626 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:28:22.290632 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.290635 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:28:22.982886 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.982792 2564 generic.go:356] "Generic (PLEG): container finished" podID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" containerID="fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb" exitCode=0 Jul 07 19:28:22.982886 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.982840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" Jul 07 19:28:22.982886 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.982878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" event={"ID":"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb","Type":"ContainerDied","Data":"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb"} Jul 07 19:28:22.983371 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.982932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl" event={"ID":"ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb","Type":"ContainerDied","Data":"071e2d5f196d13ede3af25fac02f312bc7af63fde3ab645611b770212baf1bc3"} Jul 07 19:28:22.983371 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.982954 2564 scope.go:117] "RemoveContainer" containerID="fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb" Jul 07 19:28:22.991965 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.991946 2564 scope.go:117] "RemoveContainer" containerID="fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb" Jul 07 19:28:22.992254 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:28:22.992227 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb\": container with ID starting with fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb not found: ID does not exist" containerID="fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb" Jul 07 19:28:22.992311 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:22.992265 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb"} err="failed to get container status \"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb\": rpc error: code = NotFound desc = could not find container \"fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb\": container with ID starting with fd84cb582a9128000bb0ee0454049be12690380edeb1398ed47d26056224bcbb not found: ID does not exist" Jul 07 19:28:23.003254 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:23.003221 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:28:23.007824 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:23.007795 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-6e21d5-866569959c-k4fvl"] Jul 07 19:28:24.620862 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:24.620827 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" path="/var/lib/kubelet/pods/ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb/volumes" Jul 07 19:28:24.957206 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:28:24.957114 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:29:03.932448 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:03.932409 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:29:03.933005 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:03.932689 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" containerID="cri-o://e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd" gracePeriod=30 Jul 07 19:29:05.289425 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.289382 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:29:05.289941 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.289641 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" podUID="4a474be1-e77c-4667-bf4e-7f909d932628" containerName="maas-api" containerID="cri-o://cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc" gracePeriod=30 Jul 07 19:29:05.542867 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.542799 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:29:05.623114 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.623067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca\") pod \"4a474be1-e77c-4667-bf4e-7f909d932628\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " Jul 07 19:29:05.623114 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.623121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls\") pod \"4a474be1-e77c-4667-bf4e-7f909d932628\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " Jul 07 19:29:05.623345 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.623175 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w42h\" (UniqueName: \"kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h\") pod \"4a474be1-e77c-4667-bf4e-7f909d932628\" (UID: \"4a474be1-e77c-4667-bf4e-7f909d932628\") " Jul 07 19:29:05.623513 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.623484 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a474be1-e77c-4667-bf4e-7f909d932628" (UID: "4a474be1-e77c-4667-bf4e-7f909d932628"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:29:05.625515 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.625481 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h" (OuterVolumeSpecName: "kube-api-access-4w42h") pod "4a474be1-e77c-4667-bf4e-7f909d932628" (UID: "4a474be1-e77c-4667-bf4e-7f909d932628"). InnerVolumeSpecName "kube-api-access-4w42h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:05.625515 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.625492 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a474be1-e77c-4667-bf4e-7f909d932628" (UID: "4a474be1-e77c-4667-bf4e-7f909d932628"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:29:05.724783 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.724746 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a474be1-e77c-4667-bf4e-7f909d932628-openshift-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:05.724783 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.724778 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a474be1-e77c-4667-bf4e-7f909d932628-maas-api-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:05.724783 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:05.724788 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w42h\" (UniqueName: \"kubernetes.io/projected/4a474be1-e77c-4667-bf4e-7f909d932628-kube-api-access-4w42h\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:06.205719 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.205682 2564 generic.go:356] "Generic (PLEG): container finished" podID="4a474be1-e77c-4667-bf4e-7f909d932628" containerID="cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc" exitCode=0 Jul 07 19:29:06.205900 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.205772 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" Jul 07 19:29:06.205900 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.205796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" event={"ID":"4a474be1-e77c-4667-bf4e-7f909d932628","Type":"ContainerDied","Data":"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc"} Jul 07 19:29:06.205900 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.205845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp" event={"ID":"4a474be1-e77c-4667-bf4e-7f909d932628","Type":"ContainerDied","Data":"8c2a57a59583c7bad2ca310a4b0cbe59c7a5899a970d62a9bffd4d90acd626f1"} Jul 07 19:29:06.205900 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.205868 2564 scope.go:117] "RemoveContainer" containerID="cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc" Jul 07 19:29:06.215195 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.215175 2564 scope.go:117] "RemoveContainer" containerID="cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc" Jul 07 19:29:06.215512 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:29:06.215490 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc\": container with ID starting with cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc not found: ID does not exist" containerID="cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc" Jul 07 19:29:06.215563 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.215522 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc"} err="failed to get container status \"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc\": rpc error: code = NotFound desc = could not find container \"cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc\": container with ID starting with cd9bd50ba50d047e517dc2950537121dc58427bae9d12c2fa2d13eedb8f2ebdc not found: ID does not exist" Jul 07 19:29:06.231671 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.231612 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:29:06.238466 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.238435 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-912fa0-58f7469db8-nvpwp"] Jul 07 19:29:06.620467 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:06.620435 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a474be1-e77c-4667-bf4e-7f909d932628" path="/var/lib/kubelet/pods/4a474be1-e77c-4667-bf4e-7f909d932628/volumes" Jul 07 19:29:07.733326 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:07.733278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7gmkd"] Jul 07 19:29:09.198264 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.198189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:29:09.223090 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.223058 2564 generic.go:356] "Generic (PLEG): container finished" podID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerID="e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd" exitCode=0 Jul 07 19:29:09.223242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.223116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerDied","Data":"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd"} Jul 07 19:29:09.223242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.223191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" event={"ID":"a53d9df3-888c-4af1-b025-5635c5a1865a","Type":"ContainerDied","Data":"642694aceee5beade47f088e570429f97aa9f149e6917c1010fca0de99c5e84a"} Jul 07 19:29:09.223242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.223116 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg" Jul 07 19:29:09.223242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.223211 2564 scope.go:117] "RemoveContainer" containerID="e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd" Jul 07 19:29:09.234463 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.234432 2564 scope.go:117] "RemoveContainer" containerID="dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2" Jul 07 19:29:09.243283 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.243254 2564 scope.go:117] "RemoveContainer" containerID="e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd" Jul 07 19:29:09.243613 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:29:09.243590 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd\": container with ID starting with e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd not found: ID does not exist" containerID="e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd" Jul 07 19:29:09.243731 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.243624 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd"} err="failed to get container status \"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd\": rpc error: code = NotFound desc = could not find container \"e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd\": container with ID starting with e3271cb02b575ec2d0dd7da9ae815b63cb44b1d77e71f9849f135290360383dd not found: ID does not exist" Jul 07 19:29:09.243731 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.243642 2564 scope.go:117] "RemoveContainer" containerID="dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2" Jul 07 19:29:09.243974 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:29:09.243954 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2\": container with ID starting with dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2 not found: ID does not exist" containerID="dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2" Jul 07 19:29:09.244016 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.243982 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2"} err="failed to get container status \"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2\": rpc error: code = NotFound desc = could not find container \"dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2\": container with ID starting with dc2a64ff7548c51ab8ba3e85b1ea3482a9083b75a2ba34c4b09cf0be277c72b2 not found: ID does not exist" Jul 07 19:29:09.261400 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261400 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261453 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261482 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261522 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhml7\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261556 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261595 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261644 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:09.261763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket\") pod \"a53d9df3-888c-4af1-b025-5635c5a1865a\" (UID: \"a53d9df3-888c-4af1-b025-5635c5a1865a\") " Jul 07 19:29:09.262132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261785 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:09.262132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.261971 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:29:09.262132 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262014 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:09.262272 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262171 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.262272 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262191 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.262272 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262206 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53d9df3-888c-4af1-b025-5635c5a1865a-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.262272 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262220 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.262924 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.262896 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:09.263950 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.263926 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:29:09.263950 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.263932 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:09.264100 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.264078 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:09.264186 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.264170 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7" (OuterVolumeSpecName: "kube-api-access-nhml7") pod "a53d9df3-888c-4af1-b025-5635c5a1865a" (UID: "a53d9df3-888c-4af1-b025-5635c5a1865a"). InnerVolumeSpecName "kube-api-access-nhml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:09.362963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.362913 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.362963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.362950 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.362963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.362960 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhml7\" (UniqueName: \"kubernetes.io/projected/a53d9df3-888c-4af1-b025-5635c5a1865a-kube-api-access-nhml7\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.362963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.362971 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.362963 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.362979 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53d9df3-888c-4af1-b025-5635c5a1865a-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:09.548759 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.548719 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:29:09.554312 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:09.554278 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6e21d5-openshift-default-6776c6dcf9-586mg"] Jul 07 19:29:10.620877 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:10.620840 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" path="/var/lib/kubelet/pods/a53d9df3-888c-4af1-b025-5635c5a1865a/volumes" Jul 07 19:29:11.354008 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:11.353969 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:29:11.354274 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:11.354247 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" containerID="cri-o://cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b" gracePeriod=30 Jul 07 19:29:16.618436 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.618360 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:29:16.734129 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734129 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734135 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klzlg\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734170 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734242 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734281 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734309 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734334 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy\") pod \"5b8b667b-4184-4326-b810-4ef6b4d56e04\" (UID: \"5b8b667b-4184-4326-b810-4ef6b4d56e04\") " Jul 07 19:29:16.734790 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734568 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:29:16.734790 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734688 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:16.734885 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734802 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.734885 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734821 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8b667b-4184-4326-b810-4ef6b4d56e04-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.734986 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734913 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:16.735203 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.734975 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:16.735459 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.735431 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:16.736686 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.736642 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:29:16.736780 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.736717 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg" (OuterVolumeSpecName: "kube-api-access-klzlg") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "kube-api-access-klzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:16.737287 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.737270 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:16.737287 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.737274 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b8b667b-4184-4326-b810-4ef6b4d56e04" (UID: "5b8b667b-4184-4326-b810-4ef6b4d56e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:16.835588 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835549 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835588 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835583 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835588 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835593 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835588 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835600 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835588 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835609 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835928 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835617 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzlg\" (UniqueName: \"kubernetes.io/projected/5b8b667b-4184-4326-b810-4ef6b4d56e04-kube-api-access-klzlg\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.835928 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.835625 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8b667b-4184-4326-b810-4ef6b4d56e04-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Jul 07 19:29:16.987500 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:16.987421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724180-spr8m_f4079b47-851a-403d-8356-4485e73db34d/cleanup/0.log" Jul 07 19:29:17.103722 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.103687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724195-fh82w_c4ccb445-58b0-4f7e-b8fc-19835c8e6658/cleanup/0.log" Jul 07 19:29:17.265691 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.265577 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerID="cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b" exitCode=0 Jul 07 19:29:17.265691 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.265624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerDied","Data":"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b"} Jul 07 19:29:17.265933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.265693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" event={"ID":"5b8b667b-4184-4326-b810-4ef6b4d56e04","Type":"ContainerDied","Data":"06e8ccde8ba003c24f7921cb792a7beaf7650bd818b066028085c21650c2af09"} Jul 07 19:29:17.265933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.265706 2564 scope.go:117] "RemoveContainer" containerID="cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b" Jul 07 19:29:17.265933 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.265629 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv" Jul 07 19:29:17.275851 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.275831 2564 scope.go:117] "RemoveContainer" containerID="7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83" Jul 07 19:29:17.285209 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.285184 2564 scope.go:117] "RemoveContainer" containerID="cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b" Jul 07 19:29:17.285597 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:29:17.285569 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b\": container with ID starting with cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b not found: ID does not exist" containerID="cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b" Jul 07 19:29:17.285738 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.285604 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b"} err="failed to get container status \"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b\": rpc error: code = NotFound desc = could not find container \"cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b\": container with ID starting with cc14c60490d15b47066c826c028d11b8763b8040397b02e44b354dbee271c56b not found: ID does not exist" Jul 07 19:29:17.285738 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.285629 2564 scope.go:117] "RemoveContainer" containerID="7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83" Jul 07 19:29:17.285992 ip-10-0-132-43 kubenswrapper[2564]: E0707 19:29:17.285974 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83\": container with ID starting with 7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83 not found: ID does not exist" containerID="7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83" Jul 07 19:29:17.286032 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.286000 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83"} err="failed to get container status \"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83\": rpc error: code = NotFound desc = could not find container \"7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83\": container with ID starting with 7b4a04174072b1be293c658266bb7e656fa36a9dad5c0800c4014b5abd690d83 not found: ID does not exist" Jul 07 19:29:17.289706 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.289676 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:29:17.296939 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.296910 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-912fa0-openshift-default-5f4f45dbc9-jz6qv"] Jul 07 19:29:17.690635 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.690607 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64dd56d745-jf7tv_b3de752d-ba49-4401-a36b-c78af123304c/copy-manifests/0.log" Jul 07 19:29:17.697919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:17.697896 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64dd56d745-jf7tv_b3de752d-ba49-4401-a36b-c78af123304c/manager/0.log" Jul 07 19:29:18.620983 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:18.620905 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" path="/var/lib/kubelet/pods/5b8b667b-4184-4326-b810-4ef6b4d56e04/volumes" Jul 07 19:29:19.963620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:19.963589 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7gmkd_8f50c866-4337-4b6f-8bb3-53f7bcc18446/limitador/0.log" Jul 07 19:29:20.429666 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:20.429623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jjww4_5df10e26-a133-4782-832e-41d2b14a1edd/istio-proxy/0.log" Jul 07 19:29:20.552391 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:20.552355 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qlhpb_1963bc9a-7bf0-40be-a11c-acda753d9945/discovery/0.log" Jul 07 19:29:20.897919 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:20.897886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/1.log" Jul 07 19:29:29.434518 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:29.434470 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t68ch_41b38953-6cb3-4893-ac18-51c6f63c818d/global-pull-secret-syncer/0.log" Jul 07 19:29:29.540595 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:29.540552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m7zwd_902cac6c-1abe-40c6-83f1-33f3f212dae0/konnectivity-agent/0.log" Jul 07 19:29:29.596441 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:29.596408 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-43.ec2.internal_84bdf0f4134502e8474ed50b3bda5122/haproxy/0.log" Jul 07 19:29:34.589736 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:34.589708 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7gmkd_8f50c866-4337-4b6f-8bb3-53f7bcc18446/limitador/0.log" Jul 07 19:29:36.134165 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.134072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/alertmanager/0.log" Jul 07 19:29:36.165585 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.165562 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/config-reloader/0.log" Jul 07 19:29:36.191541 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.191478 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/kube-rbac-proxy-web/0.log" Jul 07 19:29:36.217918 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.217886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/kube-rbac-proxy/0.log" Jul 07 19:29:36.242850 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.242822 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/kube-rbac-proxy-metric/0.log" Jul 07 19:29:36.274472 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.274444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/prom-label-proxy/0.log" Jul 07 19:29:36.305353 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.305312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6560098a-03e8-4fb8-b784-b48cc4f2d746/init-config-reloader/0.log" Jul 07 19:29:36.380396 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.380365 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fwtpl_3d0ee559-996b-42f8-b67b-da710033b854/kube-state-metrics/0.log" Jul 07 19:29:36.409674 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.409578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fwtpl_3d0ee559-996b-42f8-b67b-da710033b854/kube-rbac-proxy-main/0.log" Jul 07 19:29:36.439453 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.439421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fwtpl_3d0ee559-996b-42f8-b67b-da710033b854/kube-rbac-proxy-self/0.log" Jul 07 19:29:36.475068 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.475025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57bcf444-whfc9_ce0d8dca-f01f-49a3-aca1-75ea52ae578e/metrics-server/0.log" Jul 07 19:29:36.513965 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.513934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-fmgdn_364310fa-0994-4ed4-942d-3eff39767124/monitoring-plugin/0.log" Jul 07 19:29:36.661756 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.661676 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p77l6_e0b6036c-d6c9-4aab-9eaf-aa9acb046424/node-exporter/0.log" Jul 07 19:29:36.686446 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.686418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p77l6_e0b6036c-d6c9-4aab-9eaf-aa9acb046424/kube-rbac-proxy/0.log" Jul 07 19:29:36.710192 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.710167 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p77l6_e0b6036c-d6c9-4aab-9eaf-aa9acb046424/init-textfile/0.log" Jul 07 19:29:36.835251 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.835222 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pdj8m_f274a00c-eb68-44c9-bf3f-938b9b894ccb/kube-rbac-proxy-main/0.log" Jul 07 19:29:36.859461 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.859440 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pdj8m_f274a00c-eb68-44c9-bf3f-938b9b894ccb/kube-rbac-proxy-self/0.log" Jul 07 19:29:36.888414 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.888382 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pdj8m_f274a00c-eb68-44c9-bf3f-938b9b894ccb/openshift-state-metrics/0.log" Jul 07 19:29:36.936242 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.936165 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/prometheus/0.log" Jul 07 19:29:36.959847 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.959820 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/config-reloader/0.log" Jul 07 19:29:36.985186 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:36.985154 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/thanos-sidecar/0.log" Jul 07 19:29:37.011603 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.011573 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/kube-rbac-proxy-web/0.log" Jul 07 19:29:37.035791 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.035764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/kube-rbac-proxy/0.log" Jul 07 19:29:37.063620 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.063592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/kube-rbac-proxy-thanos/0.log" Jul 07 19:29:37.087007 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.086978 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dd7e1103-217e-44b6-8ed4-cb72afe4fdf9/init-config-reloader/0.log" Jul 07 19:29:37.196989 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.196905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fb65ff5f-ln2bc_508d25dd-bef1-42e8-8216-bf5df9538300/telemeter-client/0.log" Jul 07 19:29:37.222418 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.222383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fb65ff5f-ln2bc_508d25dd-bef1-42e8-8216-bf5df9538300/reload/0.log" Jul 07 19:29:37.247778 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.247745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fb65ff5f-ln2bc_508d25dd-bef1-42e8-8216-bf5df9538300/kube-rbac-proxy/0.log" Jul 07 19:29:37.282949 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.282917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/thanos-query/0.log" Jul 07 19:29:37.324329 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.324300 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/kube-rbac-proxy-web/0.log" Jul 07 19:29:37.350436 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.350402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/kube-rbac-proxy/0.log" Jul 07 19:29:37.374721 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.374691 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/prom-label-proxy/0.log" Jul 07 19:29:37.403431 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.403404 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/kube-rbac-proxy-rules/0.log" Jul 07 19:29:37.428136 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:37.428106 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dcb8865f7-fqk6t_222770a6-57c4-4ca6-b75f-e508227e5945/kube-rbac-proxy-metrics/0.log" Jul 07 19:29:39.415822 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.415783 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf"] Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416170 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416184 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416201 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416207 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416232 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" containerName="maas-api" Jul 07 19:29:39.416229 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416237 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" containerName="maas-api" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416249 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a474be1-e77c-4667-bf4e-7f909d932628" containerName="maas-api" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416257 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a474be1-e77c-4667-bf4e-7f909d932628" containerName="maas-api" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416333 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a474be1-e77c-4667-bf4e-7f909d932628" containerName="maas-api" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416348 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416357 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8b667b-4184-4326-b810-4ef6b4d56e04" containerName="istio-proxy" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416368 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2e0b7b-e4cf-4a22-a758-822c94ffa4eb" containerName="maas-api" Jul 07 19:29:39.416578 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.416379 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53d9df3-888c-4af1-b025-5635c5a1865a" containerName="istio-proxy" Jul 07 19:29:39.418740 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.418722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.421187 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.421164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5qrll\"/\"default-dockercfg-d64b8\"" Jul 07 19:29:39.421308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.421219 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qrll\"/\"openshift-service-ca.crt\"" Jul 07 19:29:39.422161 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.422138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qrll\"/\"kube-root-ca.crt\"" Jul 07 19:29:39.429982 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.429954 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf"] Jul 07 19:29:39.563143 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.563102 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-lib-modules\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.563332 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.563203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ztt\" (UniqueName: \"kubernetes.io/projected/0935515e-21b9-45cb-92d9-a89572b5cde7-kube-api-access-v4ztt\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.563332 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.563274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-podres\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.563332 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.563331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-proc\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.563438 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.563413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-sys\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.663922 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.663880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-proc\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.663941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-sys\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.663970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-lib-modules\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ztt\" (UniqueName: \"kubernetes.io/projected/0935515e-21b9-45cb-92d9-a89572b5cde7-kube-api-access-v4ztt\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664012 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-sys\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-podres\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664115 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664020 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-proc\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664327 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-lib-modules\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.664327 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.664181 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0935515e-21b9-45cb-92d9-a89572b5cde7-podres\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.674221 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.674150 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ztt\" (UniqueName: \"kubernetes.io/projected/0935515e-21b9-45cb-92d9-a89572b5cde7-kube-api-access-v4ztt\") pod \"perf-node-gather-daemonset-7fhnf\" (UID: \"0935515e-21b9-45cb-92d9-a89572b5cde7\") " pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.730211 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.730169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:39.873959 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:39.873882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf"] Jul 07 19:29:39.876227 ip-10-0-132-43 kubenswrapper[2564]: W0707 19:29:39.876192 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0935515e_21b9_45cb_92d9_a89572b5cde7.slice/crio-513955fc5105ac268577df4d01b2f34be8f8899d37c98e7295f9509100496685 WatchSource:0}: Error finding container 513955fc5105ac268577df4d01b2f34be8f8899d37c98e7295f9509100496685: Status 404 returned error can't find the container with id 513955fc5105ac268577df4d01b2f34be8f8899d37c98e7295f9509100496685 Jul 07 19:29:40.387688 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:40.387628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" event={"ID":"0935515e-21b9-45cb-92d9-a89572b5cde7","Type":"ContainerStarted","Data":"c4962bb71470bb861a29d1280cfb5b20092a6b5988d7276dcab59c434a86a5bc"} Jul 07 19:29:40.387879 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:40.387709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:40.387879 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:40.387731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" event={"ID":"0935515e-21b9-45cb-92d9-a89572b5cde7","Type":"ContainerStarted","Data":"513955fc5105ac268577df4d01b2f34be8f8899d37c98e7295f9509100496685"} Jul 07 19:29:40.421763 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:40.421714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" podStartSLOduration=1.421698351 podStartE2EDuration="1.421698351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:29:40.419398465 +0000 UTC m=+2352.362021253" watchObservedRunningTime="2026-07-07 19:29:40.421698351 +0000 UTC m=+2352.364321139" Jul 07 19:29:41.586404 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:41.586378 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnv4b_0c01a889-98bd-41d1-ad71-6d38f93af32c/dns/0.log" Jul 07 19:29:41.612493 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:41.612461 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnv4b_0c01a889-98bd-41d1-ad71-6d38f93af32c/kube-rbac-proxy/0.log" Jul 07 19:29:41.803550 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:41.803526 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pbkl2_1b298066-20c5-4300-8f0b-d3bab2a0a24c/dns-node-resolver/0.log" Jul 07 19:29:42.520441 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:42.520400 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x26db_96182dfd-28d7-4102-89e8-73269420dc2d/node-ca/0.log" Jul 07 19:29:43.503495 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:43.503470 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jjww4_5df10e26-a133-4782-832e-41d2b14a1edd/istio-proxy/0.log" Jul 07 19:29:43.626835 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:43.626804 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qlhpb_1963bc9a-7bf0-40be-a11c-acda753d9945/discovery/0.log" Jul 07 19:29:43.734308 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:43.734271 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/0.log" Jul 07 19:29:43.768126 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:43.768044 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/1.log" Jul 07 19:29:44.532502 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:44.532473 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6m8j8_ce69cba6-e743-40a2-9d41-a3ea064109a4/serve-healthcheck-canary/0.log" Jul 07 19:29:45.185122 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:45.185082 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qznkn_996859cd-ffc7-444d-b58f-ba1ec0848c19/kube-rbac-proxy/0.log" Jul 07 19:29:45.213476 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:45.213445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qznkn_996859cd-ffc7-444d-b58f-ba1ec0848c19/exporter/0.log" Jul 07 19:29:45.241128 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:45.241076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qznkn_996859cd-ffc7-444d-b58f-ba1ec0848c19/extractor/0.log" Jul 07 19:29:46.403796 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:46.403763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5qrll/perf-node-gather-daemonset-7fhnf" Jul 07 19:29:47.886662 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:47.886631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724180-spr8m_f4079b47-851a-403d-8356-4485e73db34d/cleanup/0.log" Jul 07 19:29:47.916162 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:47.916133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724195-fh82w_c4ccb445-58b0-4f7e-b8fc-19835c8e6658/cleanup/0.log" Jul 07 19:29:48.305248 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:48.305220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64dd56d745-jf7tv_b3de752d-ba49-4401-a36b-c78af123304c/manager/0.log" Jul 07 19:29:48.331261 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:48.331231 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64dd56d745-jf7tv_b3de752d-ba49-4401-a36b-c78af123304c/copy-manifests/0.log" Jul 07 19:29:49.446369 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:49.446343 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jjww4_5df10e26-a133-4782-832e-41d2b14a1edd/istio-proxy/0.log" Jul 07 19:29:49.564234 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:49.564199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qlhpb_1963bc9a-7bf0-40be-a11c-acda753d9945/discovery/0.log" Jul 07 19:29:49.662497 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:49.662465 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/0.log" Jul 07 19:29:49.681393 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:49.681342 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/1.log" Jul 07 19:29:49.844486 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:49.844460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-759f8b6db6-r8gkj_54d132bd-e691-4d90-835d-f496cafbcc82/manager/0.log" Jul 07 19:29:51.062634 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:51.062611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jjww4_5df10e26-a133-4782-832e-41d2b14a1edd/istio-proxy/0.log" Jul 07 19:29:51.182714 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:51.182682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qlhpb_1963bc9a-7bf0-40be-a11c-acda753d9945/discovery/0.log" Jul 07 19:29:51.238507 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:51.238476 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/0.log" Jul 07 19:29:51.268140 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:51.268103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7q7j_09e56f9a-26a9-4110-be3e-e277710ef5dc/istio-proxy/1.log" Jul 07 19:29:56.378669 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.378618 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/kube-multus-additional-cni-plugins/0.log" Jul 07 19:29:56.410889 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.410860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/egress-router-binary-copy/0.log" Jul 07 19:29:56.433340 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.433305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/cni-plugins/0.log" Jul 07 19:29:56.461076 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.461048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/bond-cni-plugin/0.log" Jul 07 19:29:56.489505 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.489466 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/routeoverride-cni/0.log" Jul 07 19:29:56.517539 ip-10-0-132-43 kubenswrapper[2564]: I0707 19:29:56.517508 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-69rjl_01107935-02af-4824-a87a-2d9052e0e239/whereabouts-cni-bincopy/0.log"