Jul 07 17:22:13.355136 ip-10-0-137-197 systemd[1]: Starting Kubernetes Kubelet... Jul 07 17:22:13.823067 ip-10-0-137-197 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:13.823067 ip-10-0-137-197 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 17:22:13.823067 ip-10-0-137-197 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:13.823067 ip-10-0-137-197 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 17:22:13.823067 ip-10-0-137-197 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:13.825276 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.825161 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 17:22:13.830446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830429 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.830446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830447 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830451 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830454 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830457 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830460 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830465 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830468 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830470 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830473 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830477 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830479 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830482 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830485 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830488 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830490 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830493 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830496 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830500 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830504 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830508 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.830558 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830511 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830514 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830517 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830519 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830522 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830525 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830527 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830529 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830534 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830538 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830542 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830546 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830549 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830552 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830556 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830559 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830562 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830565 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830568 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.831053 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830571 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830574 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830577 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830580 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830583 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830585 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830588 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830590 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830592 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830595 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830598 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830600 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830603 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830605 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830608 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830610 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830613 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830617 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830620 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830622 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.831511 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830625 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830627 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830630 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830632 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830635 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830637 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830639 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830643 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830646 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830649 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830652 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830655 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830657 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830660 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830662 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830665 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830667 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830670 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830673 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830675 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.832011 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830677 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830680 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830682 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830685 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830688 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830690 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830693 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830696 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830698 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830701 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830703 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830705 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830709 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830711 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830714 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830716 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830719 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830722 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.830724 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.832489 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831524 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831531 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831535 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831539 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831542 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831548 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831551 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831553 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831556 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831559 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831561 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831563 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831566 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831569 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831571 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831573 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831576 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831578 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831581 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831584 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.832959 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831586 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831589 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831592 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831594 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831597 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831600 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831602 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831605 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831607 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831610 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831613 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831615 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831618 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831620 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831625 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831628 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831632 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831634 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831640 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831643 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.833479 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831646 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831649 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831652 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831655 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831657 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831660 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831662 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831665 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831667 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831670 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831672 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831675 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831677 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831680 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831682 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831685 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831688 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831691 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831693 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.833980 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831696 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831699 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831701 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831704 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831706 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831709 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831711 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831714 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831717 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831721 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831723 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831725 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831729 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831732 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831734 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831737 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831741 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831744 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831747 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.834446 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831750 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831753 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831755 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831758 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831760 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831763 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831765 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831767 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831770 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831773 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831775 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831779 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831781 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831784 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831786 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831789 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831791 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831794 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831797 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831800 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.834934 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.831802 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832221 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832227 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832229 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832232 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832235 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832238 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832241 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832244 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832247 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832249 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832252 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832254 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832257 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832260 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832263 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832266 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832268 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832271 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832273 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.835413 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832276 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832278 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832281 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832284 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832286 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832289 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832292 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832295 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832297 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832300 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832302 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832305 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832308 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832310 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832313 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832315 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832318 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832320 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832323 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832325 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.835910 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832328 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832330 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832333 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832335 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832338 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832340 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832345 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832348 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832350 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832353 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832355 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832357 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832360 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832362 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832365 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832367 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832370 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832372 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832375 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.836386 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832378 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832381 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832383 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832386 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832388 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832391 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832393 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832396 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832398 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832401 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832403 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832407 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832411 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832415 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832418 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832421 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832423 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832426 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832428 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832432 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.836864 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832434 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832437 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832439 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832442 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832444 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832446 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832449 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832452 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832454 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832456 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832459 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832463 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832465 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832468 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832470 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832473 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832477 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832480 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832483 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.837346 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832486 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.832489 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832560 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832567 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832574 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832578 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832588 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832592 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832596 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832600 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832603 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832606 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832611 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832614 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832617 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832620 2569 flags.go:64] FLAG: --cgroup-root="" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832623 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832626 2569 flags.go:64] FLAG: --client-ca-file="" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832629 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832632 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832636 2569 flags.go:64] FLAG: --cluster-domain="" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832638 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832641 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832644 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 17:22:13.837801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832647 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832651 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832654 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832657 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832661 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832664 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832666 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832669 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832673 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832675 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832680 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832683 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832686 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832688 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832692 2569 flags.go:64] FLAG: --enable-server="true" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832695 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832700 2569 flags.go:64] FLAG: --event-burst="100" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832704 2569 flags.go:64] FLAG: --event-qps="50" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832706 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832709 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832713 2569 flags.go:64] FLAG: --eviction-hard="" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832717 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832720 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832723 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832726 2569 flags.go:64] FLAG: --eviction-soft="" Jul 07 17:22:13.838420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832728 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832731 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832734 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832737 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832740 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832742 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832745 2569 flags.go:64] FLAG: --feature-gates="" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832749 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832752 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832757 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832760 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832763 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832766 2569 flags.go:64] FLAG: --help="false" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832768 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-137-197.ec2.internal" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832771 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832774 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832777 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832781 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832784 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832786 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832789 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832792 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832795 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 17:22:13.839025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832798 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832801 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832804 2569 flags.go:64] FLAG: --kube-reserved="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832807 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832828 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832833 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832837 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832841 2569 flags.go:64] FLAG: --lock-file="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832843 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832846 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832850 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832856 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832859 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832862 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832865 2569 flags.go:64] FLAG: --logging-format="text" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832868 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832871 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832874 2569 flags.go:64] FLAG: --manifest-url="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832880 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832885 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832888 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832892 2569 flags.go:64] FLAG: --max-pods="110" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832895 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832898 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832901 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 17:22:13.839609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832904 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832907 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832910 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832913 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832920 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832923 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832926 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832929 2569 flags.go:64] FLAG: --pod-cidr="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832933 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832938 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832942 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832946 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832949 2569 flags.go:64] FLAG: --port="10250" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832952 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832955 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f58b0534c0ac295" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832958 2569 flags.go:64] FLAG: --qos-reserved="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832961 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832964 2569 flags.go:64] FLAG: --register-node="true" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832968 2569 flags.go:64] FLAG: --register-with-taints="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832972 2569 flags.go:64] FLAG: --registry-burst="10" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832975 2569 flags.go:64] FLAG: --registry-qps="5" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832978 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832980 2569 flags.go:64] FLAG: --reserved-memory="" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832984 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832987 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 17:22:13.840260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832990 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832994 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.832997 2569 flags.go:64] FLAG: --runonce="false" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833000 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833003 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833006 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833009 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833012 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833014 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833018 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833021 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833023 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833026 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833029 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833032 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833035 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833038 2569 flags.go:64] FLAG: --system-cgroups="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833041 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833048 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833051 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833054 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833057 2569 flags.go:64] FLAG: --tls-min-version="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833060 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833063 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833066 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 17:22:13.840895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833068 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833071 2569 flags.go:64] FLAG: --v="2" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833075 2569 flags.go:64] FLAG: --version="false" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833078 2569 flags.go:64] FLAG: --vmodule="" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833082 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.833085 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833230 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833234 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833238 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833241 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833243 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833246 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833248 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833252 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833254 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833257 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833259 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833263 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833266 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833269 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833271 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.841503 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833274 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833278 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833280 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833287 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833291 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833294 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833296 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833299 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833301 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833303 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833307 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833311 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833314 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833316 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833318 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833321 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833324 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833326 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833329 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833332 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.842066 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833335 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833338 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833341 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833343 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833346 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833349 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833351 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833355 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833359 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833362 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833364 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833367 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833370 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833373 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833375 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833378 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833388 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833391 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833393 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.842641 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833396 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833399 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833401 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833404 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833406 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833409 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833411 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833414 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833416 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833419 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833421 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833424 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833428 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833430 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833433 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833435 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833438 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833440 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833443 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833445 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.843289 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833448 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833450 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833453 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833456 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833458 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833461 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833463 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833466 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833468 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833473 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833476 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833479 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833481 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833484 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833487 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833490 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833493 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833495 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833498 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833500 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.844081 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833502 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833505 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833508 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833510 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.833514 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.834322 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.841654 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.841674 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.842791 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.842799 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843346 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843356 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843361 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843367 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843371 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.844609 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843382 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843387 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843392 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843397 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843402 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843406 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843413 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843419 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843424 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843429 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843434 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843438 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843443 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843452 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843456 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843461 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843465 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843469 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843474 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.844996 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843478 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843482 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843487 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843492 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843496 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843535 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843544 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843549 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843553 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843557 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843561 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843565 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843569 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843574 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843578 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843582 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843586 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843591 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843595 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843604 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.845457 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843611 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843617 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843622 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843628 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843632 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843637 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843641 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843647 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843651 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843656 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843662 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843671 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843677 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843681 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843686 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843690 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843695 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843700 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843737 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843741 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.845966 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843746 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843750 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843754 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843767 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843772 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843777 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843781 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843786 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843790 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843794 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843798 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843802 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843808 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843830 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843835 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843844 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843848 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843852 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843856 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.846459 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843860 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843864 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843868 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843872 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843876 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843881 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843885 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843889 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843894 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843902 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843907 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843911 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843948 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843952 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843957 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.846931 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.843961 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.843970 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844498 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844512 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844516 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844519 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844522 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844525 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844527 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844531 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844534 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844536 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844539 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844542 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844544 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:13.847296 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844547 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844549 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844552 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844555 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844557 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844560 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844563 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844565 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844568 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844570 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844573 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844575 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844578 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844580 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844582 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844586 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844588 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844591 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844593 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844596 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:13.847707 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844599 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844602 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844604 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844607 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844609 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844612 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844615 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844617 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844620 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844622 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844625 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844627 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844630 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844632 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844635 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844637 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844640 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844642 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844645 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844647 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:13.848203 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844650 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844652 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844655 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844658 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844660 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844663 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844665 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844668 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844670 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844673 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844676 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844679 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844681 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844684 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844686 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844689 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844691 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844694 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844696 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844699 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:13.848676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844701 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844703 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844706 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844708 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844712 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844716 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844719 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844722 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844724 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844727 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844729 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844732 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844734 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844736 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844740 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844743 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844747 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844749 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844752 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:13.849196 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844754 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844757 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844759 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844762 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844764 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844767 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:13.844770 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.844776 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.844943 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.847186 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.848829 2569 server.go:1019] "Starting client certificate rotation" Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.848924 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:22:13.849653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.848961 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:22:13.851600 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.851587 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:22:13.852793 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.852766 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:22:13.864722 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.864703 2569 log.go:25] "Validated CRI v1 runtime API" Jul 07 17:22:13.866955 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.866941 2569 log.go:25] "Validated CRI v1 image API" Jul 07 17:22:13.867855 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.867835 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 17:22:13.870598 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.870577 2569 fs.go:135] Filesystem UUIDs: map[18a360cc-c6a7-4869-9a2d-a95f245e9792:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e3e2d577-74f3-4000-8dc6-a10ae83c48d7:/dev/nvme0n1p4] Jul 07 17:22:13.870668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.870596 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 17:22:13.877490 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.877380 2569 manager.go:216] Machine: {Timestamp:2026-07-07 17:22:13.8758796 +0000 UTC m=+0.385552380 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100604 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242aae0b6eda41516f6c234e6e666d SystemUUID:ec242aae-0b6e-da41-516f-6c234e6e666d BootID:a009aa50-a5e7-4fb5-9f48-8d18652a6f2f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:25:7b:c7:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:25:7b:c7:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:90:99:6c:2a:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 17:22:13.877490 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.877483 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 17:22:13.877606 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.877566 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 17:22:13.880959 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.880923 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 17:22:13.881099 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.880961 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 17:22:13.882425 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.882414 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 17:22:13.882460 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.882429 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 17:22:13.882460 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.882436 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:22:13.883073 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.883059 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:22:13.884481 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.884469 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 17:22:13.887486 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887470 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:22:13.887662 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887650 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:22:13.887797 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887787 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 17:22:13.887845 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887836 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 07 17:22:13.887878 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887857 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 17:22:13.887905 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887881 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 17:22:13.887932 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887906 2569 kubelet.go:398] "Adding apiserver pod source" Jul 07 17:22:13.887932 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.887918 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 17:22:13.888518 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.888489 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:22:13.888586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.888538 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:22:13.892021 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.892003 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 17:22:13.894312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.894296 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 17:22:13.894389 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.894335 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 17:22:13.895565 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895550 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895572 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895582 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895590 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895603 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895611 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895624 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895633 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895643 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895651 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895663 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 17:22:13.895680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.895671 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 17:22:13.896524 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.896512 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 17:22:13.896577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.896527 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 17:22:13.899743 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.899728 2569 server.go:1268] "Started kubelet" Jul 07 17:22:13.899861 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.899800 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 17:22:13.900023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.899973 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 17:22:13.900092 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.900079 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 17:22:13.900411 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.900381 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 17:22:13.900625 ip-10-0-137-197 systemd[1]: Started Kubernetes Kubelet. Jul 07 17:22:13.900763 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.900606 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 17:22:13.900763 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.900751 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 17:22:13.901714 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.901702 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 07 17:22:13.901945 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.901930 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 17:22:13.905450 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.905432 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 17:22:13.908203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.908184 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 17:22:13.908309 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.908203 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 17:22:13.908420 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.908400 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:13.908960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.908940 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 17:22:13.909040 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.908965 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 17:22:13.909040 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.908943 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 17:22:13.909146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909057 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 07 17:22:13.909146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909064 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 07 17:22:13.909520 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909505 2569 factory.go:153] Registering CRI-O factory Jul 07 17:22:13.909595 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909546 2569 factory.go:223] Registration of the crio container factory successfully Jul 07 17:22:13.909658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909596 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 17:22:13.909658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909605 2569 factory.go:55] Registering systemd factory Jul 07 17:22:13.909658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909613 2569 factory.go:223] Registration of the systemd container factory successfully Jul 07 17:22:13.909658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909635 2569 factory.go:103] Registering Raw factory Jul 07 17:22:13.909658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.909649 2569 manager.go:1196] Started watching for new ooms in manager Jul 07 17:22:13.910469 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.910374 2569 manager.go:349] Starting recovery of all containers Jul 07 17:22:13.912615 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.912582 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 17:22:13.912766 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.912743 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 17:22:13.914207 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.912668 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-197.ec2.internal.18c011b7e18fcf28 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-197.ec2.internal,UID:ip-10-0-137-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-197.ec2.internal,},FirstTimestamp:2026-07-07 17:22:13.899702056 +0000 UTC m=+0.409374835,LastTimestamp:2026-07-07 17:22:13.899702056 +0000 UTC m=+0.409374835,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-197.ec2.internal,}" Jul 07 17:22:13.923560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.923539 2569 manager.go:354] Recovery completed Jul 07 17:22:13.927876 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.927864 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:13.930258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930236 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:13.930320 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930273 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:13.930320 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930286 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:13.930850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930830 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:13.930895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930829 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 17:22:13.930895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930870 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 17:22:13.930895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.930887 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:22:13.931985 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.931973 2569 policy_none.go:49] "None policy: Start" Jul 07 17:22:13.932023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.931989 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 17:22:13.932023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.931999 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 17:22:13.933180 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.933162 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:13.933180 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.933177 2569 policy_none.go:47] "Start" Jul 07 17:22:13.933271 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.933198 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:13.933271 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.933212 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:13.934276 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.934104 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-197.ec2.internal.18c011b7e3621021 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-197.ec2.internal,UID:ip-10-0-137-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-197.ec2.internal,},FirstTimestamp:2026-07-07 17:22:13.930258465 +0000 UTC m=+0.439931246,LastTimestamp:2026-07-07 17:22:13.930258465 +0000 UTC m=+0.439931246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-197.ec2.internal,}" Jul 07 17:22:13.945769 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.945691 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-197.ec2.internal.18c011b7e362658f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-197.ec2.internal,UID:ip-10-0-137-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-197.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-197.ec2.internal,},FirstTimestamp:2026-07-07 17:22:13.930280335 +0000 UTC m=+0.439953114,LastTimestamp:2026-07-07 17:22:13.930280335 +0000 UTC m=+0.439953114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-197.ec2.internal,}" Jul 07 17:22:13.950643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.950614 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5w9b" Jul 07 17:22:13.958133 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.958055 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-197.ec2.internal.18c011b7e3628c3c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-197.ec2.internal,UID:ip-10-0-137-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-197.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-197.ec2.internal,},FirstTimestamp:2026-07-07 17:22:13.930290236 +0000 UTC m=+0.439963014,LastTimestamp:2026-07-07 17:22:13.930290236 +0000 UTC m=+0.439963014,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-197.ec2.internal,}" Jul 07 17:22:13.962042 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.962025 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5w9b" Jul 07 17:22:13.967483 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967469 2569 manager.go:337] "Starting Device Plugin manager" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.967499 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967509 2569 server.go:87] "Starting device plugin registration server" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967771 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967785 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967878 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967957 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:13.967965 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.968593 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 17:22:13.981203 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:13.968644 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.053830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.053789 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 17:22:14.055139 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.055121 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 17:22:14.055194 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.055172 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 17:22:14.055228 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.055205 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 17:22:14.055271 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.055245 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 17:22:14.058192 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.058086 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:14.068397 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.068378 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:14.069637 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.069623 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:14.069701 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.069658 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:14.069701 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.069673 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:14.069701 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.069695 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.081412 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.081373 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.081412 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.081394 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-197.ec2.internal\": node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.117413 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.117391 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.156346 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.156322 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal"] Jul 07 17:22:14.156412 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.156398 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:14.158749 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.158726 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:14.158749 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.158754 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:14.158904 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.158763 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:14.160125 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160113 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:14.160227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160214 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.160280 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160272 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:14.160942 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160923 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:14.161024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160949 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:14.161024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160952 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:14.161024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160959 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:14.161024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160970 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:14.161024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.160990 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:14.162268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.162255 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.162317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.162279 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:14.163020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.163001 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:14.163088 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.163031 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:14.163088 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.163045 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:14.185534 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.185513 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-197.ec2.internal\" not found" node="ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.190217 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.190203 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-197.ec2.internal\" not found" node="ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.218191 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.218160 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.310241 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.310210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.310376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.310240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.310376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.310274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad89d569afe56b30feba4f95595fae64-config\") pod \"kube-apiserver-proxy-ip-10-0-137-197.ec2.internal\" (UID: \"ad89d569afe56b30feba4f95595fae64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.318235 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.318214 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.410976 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.410910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad89d569afe56b30feba4f95595fae64-config\") pod \"kube-apiserver-proxy-ip-10-0-137-197.ec2.internal\" (UID: \"ad89d569afe56b30feba4f95595fae64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.410976 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.410937 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad89d569afe56b30feba4f95595fae64-config\") pod \"kube-apiserver-proxy-ip-10-0-137-197.ec2.internal\" (UID: \"ad89d569afe56b30feba4f95595fae64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.410976 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.410972 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.411148 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.410947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.411148 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.411016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.411148 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.411049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30e6115e2341600f25267a786c8ff77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal\" (UID: \"e30e6115e2341600f25267a786c8ff77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.419016 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.418993 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.489136 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.489105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.493654 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.493459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:14.519857 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.519836 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.620353 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.620323 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.720874 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.720785 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.821305 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.821275 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.848462 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.848425 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 17:22:14.848922 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.848565 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:22:14.921698 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:14.921675 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:14.965407 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.965371 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:17:15 +0000 UTC" deadline="2027-12-23 14:33:49.766032203 +0000 UTC" Jul 07 17:22:14.965407 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:14.965405 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12813h11m34.80063016s" Jul 07 17:22:15.022154 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:15.022051 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:15.032864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:15.032843 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:15.047771 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:15.047740 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad89d569afe56b30feba4f95595fae64.slice/crio-deebcb810c2350387c6c2d366e5d60505dfce43eda380c8196fb0bb22628c6f5 WatchSource:0}: Error finding container deebcb810c2350387c6c2d366e5d60505dfce43eda380c8196fb0bb22628c6f5: Status 404 returned error can't find the container with id deebcb810c2350387c6c2d366e5d60505dfce43eda380c8196fb0bb22628c6f5 Jul 07 17:22:15.048093 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:15.048078 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30e6115e2341600f25267a786c8ff77.slice/crio-4e365830442a34d97c48b89b745ef38d23f9ac9e5ea1f8946eb27bb5fa8f1334 WatchSource:0}: Error finding container 4e365830442a34d97c48b89b745ef38d23f9ac9e5ea1f8946eb27bb5fa8f1334: Status 404 returned error can't find the container with id 4e365830442a34d97c48b89b745ef38d23f9ac9e5ea1f8946eb27bb5fa8f1334 Jul 07 17:22:15.053583 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:15.053566 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:22:15.058164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:15.058124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" event={"ID":"e30e6115e2341600f25267a786c8ff77","Type":"ContainerStarted","Data":"4e365830442a34d97c48b89b745ef38d23f9ac9e5ea1f8946eb27bb5fa8f1334"} Jul 07 17:22:15.059134 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:15.059115 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" event={"ID":"ad89d569afe56b30feba4f95595fae64","Type":"ContainerStarted","Data":"deebcb810c2350387c6c2d366e5d60505dfce43eda380c8196fb0bb22628c6f5"} Jul 07 17:22:15.122498 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:15.122470 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:16.531083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.531064 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:16.532105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.532091 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:16.532173 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.532118 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:16.532173 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.532127 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:16.612922 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:16.612894 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:16.713501 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:16.713422 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:16.725626 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.725605 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:16.813786 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:16.813756 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-197.ec2.internal\" not found" Jul 07 17:22:16.882602 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.882579 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:16.898880 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.898852 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" Jul 07 17:22:16.912355 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.912337 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:22:16.913592 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.913579 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" Jul 07 17:22:16.926700 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:16.926681 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:22:17.069570 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.069503 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:17.278525 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.278491 2569 apiserver.go:52] "Watching apiserver" Jul 07 17:22:17.286701 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.286678 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 17:22:17.287058 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.287037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h8x4x","kube-system/konnectivity-agent-pbt7g","kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw","openshift-cluster-node-tuning-operator/tuned-4n2ft","openshift-dns/node-resolver-qkd9s","openshift-image-registry/node-ca-hnzsh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal","openshift-multus/multus-5b2qt","openshift-multus/multus-additional-cni-plugins-pbqfz","openshift-multus/network-metrics-daemon-9sfzm","openshift-network-diagnostics/network-check-target-88r7j","openshift-network-operator/iptables-alerter-6whmp","openshift-ovn-kubernetes/ovnkube-node-f9nxj"] Jul 07 17:22:17.288593 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.288577 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.289712 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.289680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.290787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.290767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.291906 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.291881 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.292159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.292258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292216 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.292258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292229 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.292258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292254 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8bl4t\"" Jul 07 17:22:17.292437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292398 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 17:22:17.292570 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b2zj8\"" Jul 07 17:22:17.292667 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.292579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 17:22:17.293022 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.293009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.293390 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.293370 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 17:22:17.293476 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.293392 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 17:22:17.293476 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.293437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.293666 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.293653 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 17:22:17.294085 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.294067 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hx5rw\"" Jul 07 17:22:17.294085 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.294076 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.294227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.294099 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 17:22:17.294227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.294079 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.294227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.294209 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.295127 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.295109 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 17:22:17.295335 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.295320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.295491 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.295472 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.295682 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.295662 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.295773 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.295695 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tvd89\"" Jul 07 17:22:17.296111 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.296058 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8mkqb\"" Jul 07 17:22:17.296253 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.296233 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.296579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.296561 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.296737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.296719 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gww52\"" Jul 07 17:22:17.296807 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.296774 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.297196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.297180 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.298305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.298263 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 17:22:17.298393 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.298359 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.298454 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.298424 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:17.299773 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.299755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.302495 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.302471 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzfxc\"" Jul 07 17:22:17.302788 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.302768 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 17:22:17.302873 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.302831 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 17:22:17.303201 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.303185 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 17:22:17.303352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.303335 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 17:22:17.304862 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.303938 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 17:22:17.304862 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.304718 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5n7xp\"" Jul 07 17:22:17.305253 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.305235 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 17:22:17.306166 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.306148 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.307587 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.307565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.307688 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.307649 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:17.308759 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.308740 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:17.308839 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.308802 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:17.308936 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.308906 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pkmrd\"" Jul 07 17:22:17.308936 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.308923 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 17:22:17.309143 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.309130 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-etc-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-etc-kubernetes\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt5v\" (UniqueName: \"kubernetes.io/projected/8addf0d9-b1e2-482f-a378-778455447ccf-kube-api-access-5dt5v\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-systemd-units\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-node-log\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9m8\" (UniqueName: \"kubernetes.io/projected/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-kube-api-access-rm9m8\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.310474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-system-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-multus-certs\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310499 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-script-lib\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbmw\" (UniqueName: \"kubernetes.io/projected/dde65482-9be3-4c80-8e60-d9900979c28b-kube-api-access-5lbmw\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eaed74-372b-4f2d-a80e-bfb1b0398601-host\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-os-release\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-kubelet\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6pv\" (UniqueName: \"kubernetes.io/projected/1739a7b0-6920-4604-aa9d-939f5c92e46b-kube-api-access-rk6pv\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgck\" (UniqueName: \"kubernetes.io/projected/e009614e-0bb8-4458-9c73-33a9268f90f3-kube-api-access-8qgck\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-systemd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-netd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-device-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-tmp\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.310781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-k8s-cni-cncf-io\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-multus\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-netns\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-bin\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310890 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovn-node-metrics-cert\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-host\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-bin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-run\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.310989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-sys\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-registration-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311111 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-sys-fs\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66ff5915-cd61-45ea-8199-1b41e636c35f-konnectivity-ca\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75eaed74-372b-4f2d-a80e-bfb1b0398601-serviceca\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.311268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59h8\" (UniqueName: \"kubernetes.io/projected/75eaed74-372b-4f2d-a80e-bfb1b0398601-kube-api-access-r59h8\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-cnibin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8addf0d9-b1e2-482f-a378-778455447ccf-host-slash\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311415 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-socket-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5wn\" (UniqueName: \"kubernetes.io/projected/595c820e-0463-4011-851d-f3dea5f189e6-kube-api-access-4z5wn\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-kubernetes\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-tuned\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-kubelet\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311560 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-kubelet-config\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-conf\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66ff5915-cd61-45ea-8199-1b41e636c35f-agent-certs\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.311736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-cni-binary-copy\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-socket-dir-parent\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-conf-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-log-socket\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-config\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-var-lib-kubelet\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-netns\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311842 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-env-overrides\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-modprobe-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-daemon-config\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-dbus\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8addf0d9-b1e2-482f-a378-778455447ccf-iptables-alerter-script\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-ovn\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.311989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/595c820e-0463-4011-851d-f3dea5f189e6-hosts-file\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-lib-modules\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.312402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysconfig\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-systemd\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-var-lib-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595c820e-0463-4011-851d-f3dea5f189e6-tmp-dir\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-hostroot\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-slash\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.313045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.312569 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:22:17.394964 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.394939 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-894cf" Jul 07 17:22:17.399369 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.399347 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 17:22:17.403196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.403176 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-894cf" Jul 07 17:22:17.412562 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-system-cni-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.412663 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.412663 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.412663 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.412844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.412844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412677 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.412844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412692 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66ff5915-cd61-45ea-8199-1b41e636c35f-agent-certs\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.412844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-cni-binary-copy\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.412844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412778 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412843 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-socket-dir-parent\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-conf-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412928 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-log-socket\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-config\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-var-lib-kubelet\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.412991 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-conf-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413008 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 17:22:17.413082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-netns\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-log-socket\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413138 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-socket-dir-parent\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-netns\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-env-overrides\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413199 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-modprobe-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413229 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-daemon-config\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-dbus\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8addf0d9-b1e2-482f-a378-778455447ccf-iptables-alerter-script\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-ovn\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-cni-binary-copy\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/595c820e-0463-4011-851d-f3dea5f189e6-hosts-file\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-lib-modules\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysconfig\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-systemd\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-var-lib-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.413437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595c820e-0463-4011-851d-f3dea5f189e6-tmp-dir\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-hostroot\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413472 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-cnibin\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413506 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-slash\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-etc-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413529 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-config\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-etc-kubernetes\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kv7f\" (UniqueName: \"kubernetes.io/projected/2e87ea91-61dd-48f2-a238-5bc98e293851-kube-api-access-5kv7f\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt5v\" (UniqueName: \"kubernetes.io/projected/8addf0d9-b1e2-482f-a378-778455447ccf-kube-api-access-5dt5v\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-var-lib-kubelet\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-systemd-units\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-node-log\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9m8\" (UniqueName: \"kubernetes.io/projected/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-kube-api-access-rm9m8\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-system-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-multus-certs\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.414300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-script-lib\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbmw\" (UniqueName: \"kubernetes.io/projected/dde65482-9be3-4c80-8e60-d9900979c28b-kube-api-access-5lbmw\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eaed74-372b-4f2d-a80e-bfb1b0398601-host\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-os-release\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413740 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-kubelet\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413756 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-daemon-config\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6pv\" (UniqueName: \"kubernetes.io/projected/1739a7b0-6920-4604-aa9d-939f5c92e46b-kube-api-access-rk6pv\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgck\" (UniqueName: \"kubernetes.io/projected/e009614e-0bb8-4458-9c73-33a9268f90f3-kube-api-access-8qgck\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-os-release\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595c820e-0463-4011-851d-f3dea5f189e6-tmp-dir\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-systemd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.413841 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-netd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-device-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.413900 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:17.913883399 +0000 UTC m=+3.034087726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:17.415046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413917 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-device-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-tmp\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413949 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-hostroot\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-k8s-cni-cncf-io\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-multus\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.413999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-slash\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414012 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-multus\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414017 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-dbus\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-netns\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414083 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-bin\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414091 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-modprobe-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovn-node-metrics-cert\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-host\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-d\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-ovn\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414224 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/595c820e-0463-4011-851d-f3dea5f189e6-hosts-file\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-lib-modules\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-env-overrides\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414407 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysconfig\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-bin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgsv\" (UniqueName: \"kubernetes.io/projected/ca3aed3c-1f0d-466a-834e-86aa02f6e884-kube-api-access-ppgsv\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414458 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-systemd\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-k8s-cni-cncf-io\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8addf0d9-b1e2-482f-a378-778455447ccf-iptables-alerter-script\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-run\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-var-lib-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414555 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-run\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414606 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-host\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-netns\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-etc-openvswitch\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-etc-kubernetes\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-sys\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-registration-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414801 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-systemd-units\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414893 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-cni-bin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.416938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eaed74-372b-4f2d-a80e-bfb1b0398601-host\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-sys-fs\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414930 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-kubelet\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66ff5915-cd61-45ea-8199-1b41e636c35f-konnectivity-ca\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-node-log\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414951 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-registration-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75eaed74-372b-4f2d-a80e-bfb1b0398601-serviceca\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.414987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59h8\" (UniqueName: \"kubernetes.io/projected/75eaed74-372b-4f2d-a80e-bfb1b0398601-kube-api-access-r59h8\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-os-release\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415013 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-cnibin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8addf0d9-b1e2-482f-a378-778455447ccf-host-slash\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415053 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-system-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-socket-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5wn\" (UniqueName: \"kubernetes.io/projected/595c820e-0463-4011-851d-f3dea5f189e6-kube-api-access-4z5wn\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-sys\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415105 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-run-multus-certs\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-kubernetes\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-tuned\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.417772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-kubelet\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-kubelet-config\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-conf\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-socket-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415617 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-kubernetes\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-bin\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415798 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66ff5915-cd61-45ea-8199-1b41e636c35f-konnectivity-ca\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415916 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75eaed74-372b-4f2d-a80e-bfb1b0398601-serviceca\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415967 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8addf0d9-b1e2-482f-a378-778455447ccf-host-slash\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.415995 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-cnibin\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-cni-netd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dde65482-9be3-4c80-8e60-d9900979c28b-sys-fs\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416070 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-host-var-lib-kubelet\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416088 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-run-systemd\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.418801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e009614e-0bb8-4458-9c73-33a9268f90f3-multus-cni-dir\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.419608 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416165 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a7b0-6920-4604-aa9d-939f5c92e46b-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.419608 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416166 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87d5a160-779e-4370-b6fb-96a892b2dbca-kubelet-config\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.419608 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-sysctl-conf\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.419608 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.416589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovnkube-script-lib\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.419608 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.418698 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66ff5915-cd61-45ea-8199-1b41e636c35f-agent-certs\") pod \"konnectivity-agent-pbt7g\" (UID: \"66ff5915-cd61-45ea-8199-1b41e636c35f\") " pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.420167 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.420149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-tmp\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.420839 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.420628 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-etc-tuned\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.422300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.422182 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1739a7b0-6920-4604-aa9d-939f5c92e46b-ovn-node-metrics-cert\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.429936 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.429913 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbmw\" (UniqueName: \"kubernetes.io/projected/dde65482-9be3-4c80-8e60-d9900979c28b-kube-api-access-5lbmw\") pod \"aws-ebs-csi-driver-node-m9vnw\" (UID: \"dde65482-9be3-4c80-8e60-d9900979c28b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.430810 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.430784 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5wn\" (UniqueName: \"kubernetes.io/projected/595c820e-0463-4011-851d-f3dea5f189e6-kube-api-access-4z5wn\") pod \"node-resolver-qkd9s\" (UID: \"595c820e-0463-4011-851d-f3dea5f189e6\") " pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.431323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.431301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt5v\" (UniqueName: \"kubernetes.io/projected/8addf0d9-b1e2-482f-a378-778455447ccf-kube-api-access-5dt5v\") pod \"iptables-alerter-6whmp\" (UID: \"8addf0d9-b1e2-482f-a378-778455447ccf\") " pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.431420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.431335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9m8\" (UniqueName: \"kubernetes.io/projected/4efe1724-f3ab-417d-bfdc-98ce7f4019e8-kube-api-access-rm9m8\") pod \"tuned-4n2ft\" (UID: \"4efe1724-f3ab-417d-bfdc-98ce7f4019e8\") " pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.431420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.431406 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59h8\" (UniqueName: \"kubernetes.io/projected/75eaed74-372b-4f2d-a80e-bfb1b0398601-kube-api-access-r59h8\") pod \"node-ca-hnzsh\" (UID: \"75eaed74-372b-4f2d-a80e-bfb1b0398601\") " pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.432305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.432265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6pv\" (UniqueName: \"kubernetes.io/projected/1739a7b0-6920-4604-aa9d-939f5c92e46b-kube-api-access-rk6pv\") pod \"ovnkube-node-f9nxj\" (UID: \"1739a7b0-6920-4604-aa9d-939f5c92e46b\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.433410 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.433336 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgck\" (UniqueName: \"kubernetes.io/projected/e009614e-0bb8-4458-9c73-33a9268f90f3-kube-api-access-8qgck\") pod \"multus-5b2qt\" (UID: \"e009614e-0bb8-4458-9c73-33a9268f90f3\") " pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.515698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-cnibin\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kv7f\" (UniqueName: \"kubernetes.io/projected/2e87ea91-61dd-48f2-a238-5bc98e293851-kube-api-access-5kv7f\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-cnibin\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515839 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-os-release\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.515929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgsv\" (UniqueName: \"kubernetes.io/projected/ca3aed3c-1f0d-466a-834e-86aa02f6e884-kube-api-access-ppgsv\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.515947 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.515980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.516007 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:18.015986779 +0000 UTC m=+3.136191102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-system-cni-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516244 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-os-release\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516305 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516341 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516453 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e87ea91-61dd-48f2-a238-5bc98e293851-system-cni-dir\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.516661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.516560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e87ea91-61dd-48f2-a238-5bc98e293851-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.524561 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.524532 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:17.524561 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.524553 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:17.524561 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.524567 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:17.524780 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.524638 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:18.024621794 +0000 UTC m=+3.144826105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:17.526323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.526278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kv7f\" (UniqueName: \"kubernetes.io/projected/2e87ea91-61dd-48f2-a238-5bc98e293851-kube-api-access-5kv7f\") pod \"multus-additional-cni-plugins-pbqfz\" (UID: \"2e87ea91-61dd-48f2-a238-5bc98e293851\") " pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.526594 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.526577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgsv\" (UniqueName: \"kubernetes.io/projected/ca3aed3c-1f0d-466a-834e-86aa02f6e884-kube-api-access-ppgsv\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:17.599398 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.599319 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnzsh" Jul 07 17:22:17.610215 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.610177 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eaed74_372b_4f2d_a80e_bfb1b0398601.slice/crio-074f603c2cb1d0dd1c566263cf0dc719e7bea92370495bcaf14d9136761141b3 WatchSource:0}: Error finding container 074f603c2cb1d0dd1c566263cf0dc719e7bea92370495bcaf14d9136761141b3: Status 404 returned error can't find the container with id 074f603c2cb1d0dd1c566263cf0dc719e7bea92370495bcaf14d9136761141b3 Jul 07 17:22:17.611030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.611010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6whmp" Jul 07 17:22:17.619680 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.619651 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8addf0d9_b1e2_482f_a378_778455447ccf.slice/crio-939b0bf6a602916b5a10babb9aac9dd36e3a8e5d3e37efa7fb6d4a2b6b610a1a WatchSource:0}: Error finding container 939b0bf6a602916b5a10babb9aac9dd36e3a8e5d3e37efa7fb6d4a2b6b610a1a: Status 404 returned error can't find the container with id 939b0bf6a602916b5a10babb9aac9dd36e3a8e5d3e37efa7fb6d4a2b6b610a1a Jul 07 17:22:17.620897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.620878 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:17.629789 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.629765 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739a7b0_6920_4604_aa9d_939f5c92e46b.slice/crio-80913d3d16f892cc7dde3f9ce585302521d67e90b23aede5b4d5b3307e18497f WatchSource:0}: Error finding container 80913d3d16f892cc7dde3f9ce585302521d67e90b23aede5b4d5b3307e18497f: Status 404 returned error can't find the container with id 80913d3d16f892cc7dde3f9ce585302521d67e90b23aede5b4d5b3307e18497f Jul 07 17:22:17.632884 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.632765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" Jul 07 17:22:17.640427 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.640256 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" Jul 07 17:22:17.641603 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.641481 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde65482_9be3_4c80_8e60_d9900979c28b.slice/crio-257f686e59707db94774117b829ceaf5721547d1628994f02641bff635a045dc WatchSource:0}: Error finding container 257f686e59707db94774117b829ceaf5721547d1628994f02641bff635a045dc: Status 404 returned error can't find the container with id 257f686e59707db94774117b829ceaf5721547d1628994f02641bff635a045dc Jul 07 17:22:17.647873 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.647852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkd9s" Jul 07 17:22:17.655342 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.655317 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe1724_f3ab_417d_bfdc_98ce7f4019e8.slice/crio-a9fcfdccca46c8587c3cb5a40162772f4e932840b8d643a69f031ef30158795d WatchSource:0}: Error finding container a9fcfdccca46c8587c3cb5a40162772f4e932840b8d643a69f031ef30158795d: Status 404 returned error can't find the container with id a9fcfdccca46c8587c3cb5a40162772f4e932840b8d643a69f031ef30158795d Jul 07 17:22:17.656195 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.656180 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b2qt" Jul 07 17:22:17.658032 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.657997 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595c820e_0463_4011_851d_f3dea5f189e6.slice/crio-4fb0fadc7555e2b9ba7be4477c2c167feed87fc73c479873190f0408cb638fb7 WatchSource:0}: Error finding container 4fb0fadc7555e2b9ba7be4477c2c167feed87fc73c479873190f0408cb638fb7: Status 404 returned error can't find the container with id 4fb0fadc7555e2b9ba7be4477c2c167feed87fc73c479873190f0408cb638fb7 Jul 07 17:22:17.663399 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.663332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:17.665675 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.665644 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode009614e_0bb8_4458_9c73_33a9268f90f3.slice/crio-46827b4e15ea373ffba18c325ff67e019afcb20c368d6a2674e09f251e19b625 WatchSource:0}: Error finding container 46827b4e15ea373ffba18c325ff67e019afcb20c368d6a2674e09f251e19b625: Status 404 returned error can't find the container with id 46827b4e15ea373ffba18c325ff67e019afcb20c368d6a2674e09f251e19b625 Jul 07 17:22:17.669231 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.669065 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" Jul 07 17:22:17.673877 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:17.673853 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ff5915_cd61_45ea_8199_1b41e636c35f.slice/crio-abf739ad53bbf948875c9d95f9773ee3b02f9edf61a133d3bfad62cf76d97f4f WatchSource:0}: Error finding container abf739ad53bbf948875c9d95f9773ee3b02f9edf61a133d3bfad62cf76d97f4f: Status 404 returned error can't find the container with id abf739ad53bbf948875c9d95f9773ee3b02f9edf61a133d3bfad62cf76d97f4f Jul 07 17:22:17.919393 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:17.919314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:17.919516 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.919468 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:17.919569 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:17.919532 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:18.919517046 +0000 UTC m=+4.039721354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:18.020470 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.020421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:18.020621 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.020581 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:18.020678 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.020640 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:19.02062582 +0000 UTC m=+4.140830131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:18.120880 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.120840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:18.121055 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.121010 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:18.121055 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.121037 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:18.121055 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.121048 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:18.121216 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.121109 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:19.121091908 +0000 UTC m=+4.241296219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:18.405069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.404979 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:17:17 +0000 UTC" deadline="2028-02-25 23:43:14.919202802 +0000 UTC" Jul 07 17:22:18.405069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.405013 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14358h20m56.51419319s" Jul 07 17:22:18.445737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.445709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:18.445895 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.445833 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:18.446015 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.445894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:18.446015 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.445998 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:18.456524 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.456491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerStarted","Data":"3bdc9cf8447144ab4f26ae03cf48cdc0181521f7d124cb2cd20d180f3a270f7e"} Jul 07 17:22:18.457682 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.457644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pbt7g" event={"ID":"66ff5915-cd61-45ea-8199-1b41e636c35f","Type":"ContainerStarted","Data":"abf739ad53bbf948875c9d95f9773ee3b02f9edf61a133d3bfad62cf76d97f4f"} Jul 07 17:22:18.459150 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.459126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkd9s" event={"ID":"595c820e-0463-4011-851d-f3dea5f189e6","Type":"ContainerStarted","Data":"4fb0fadc7555e2b9ba7be4477c2c167feed87fc73c479873190f0408cb638fb7"} Jul 07 17:22:18.460422 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.460399 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" event={"ID":"4efe1724-f3ab-417d-bfdc-98ce7f4019e8","Type":"ContainerStarted","Data":"a9fcfdccca46c8587c3cb5a40162772f4e932840b8d643a69f031ef30158795d"} Jul 07 17:22:18.461646 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.461622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" event={"ID":"dde65482-9be3-4c80-8e60-d9900979c28b","Type":"ContainerStarted","Data":"257f686e59707db94774117b829ceaf5721547d1628994f02641bff635a045dc"} Jul 07 17:22:18.463066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.463008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6whmp" event={"ID":"8addf0d9-b1e2-482f-a378-778455447ccf","Type":"ContainerStarted","Data":"939b0bf6a602916b5a10babb9aac9dd36e3a8e5d3e37efa7fb6d4a2b6b610a1a"} Jul 07 17:22:18.464347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.464305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnzsh" event={"ID":"75eaed74-372b-4f2d-a80e-bfb1b0398601","Type":"ContainerStarted","Data":"074f603c2cb1d0dd1c566263cf0dc719e7bea92370495bcaf14d9136761141b3"} Jul 07 17:22:18.466909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.466874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b2qt" event={"ID":"e009614e-0bb8-4458-9c73-33a9268f90f3","Type":"ContainerStarted","Data":"46827b4e15ea373ffba18c325ff67e019afcb20c368d6a2674e09f251e19b625"} Jul 07 17:22:18.468415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.468354 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"80913d3d16f892cc7dde3f9ce585302521d67e90b23aede5b4d5b3307e18497f"} Jul 07 17:22:18.928606 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:18.928571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:18.928771 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.928748 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:18.928855 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:18.928834 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:20.928812801 +0000 UTC m=+6.049017134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:19.029787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:19.029752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:19.029967 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.029918 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:19.030029 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.029996 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:21.029974978 +0000 UTC m=+6.150179285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:19.130486 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:19.130377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:19.130656 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.130576 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:19.130656 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.130595 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:19.130656 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.130611 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:19.130818 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.130669 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:21.130650717 +0000 UTC m=+6.250855043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:19.406253 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:19.405859 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:17:17 +0000 UTC" deadline="2028-02-06 03:16:40.489757371 +0000 UTC" Jul 07 17:22:19.406253 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:19.405919 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13881h54m21.083842062s" Jul 07 17:22:19.447815 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:19.447781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:19.448068 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:19.448042 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:20.445381 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:20.445342 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:20.445833 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:20.445484 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:20.445894 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:20.445858 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:20.445974 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:20.445950 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:20.949096 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:20.949058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:20.949268 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:20.949226 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:20.949348 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:20.949306 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:24.949270002 +0000 UTC m=+10.069474314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:21.050348 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:21.050313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:21.050522 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.050467 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:21.050522 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.050513 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:25.050500837 +0000 UTC m=+10.170705145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:21.151242 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:21.151201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:21.151449 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.151435 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:21.151518 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.151455 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:21.151518 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.151468 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:21.151629 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.151526 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:25.151507531 +0000 UTC m=+10.271711855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:21.446177 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:21.445887 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:21.446177 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:21.446024 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:22.446203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:22.445729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:22.446203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:22.445773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:22.446203 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:22.445860 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:22.446203 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:22.446003 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:23.445553 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:23.445523 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:23.445735 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:23.445660 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:24.445983 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:24.445948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:24.446478 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:24.446085 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:24.446478 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:24.445948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:24.446595 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:24.446539 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:24.979567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:24.979524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:24.979735 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:24.979694 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:24.979800 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:24.979770 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:32.979749026 +0000 UTC m=+18.099953337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:25.080630 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:25.080535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:25.080782 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.080718 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:25.080852 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.080792 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:33.080771643 +0000 UTC m=+18.200975954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:25.182003 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:25.181896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:25.182174 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.182082 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:25.182174 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.182105 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:25.182174 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.182119 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:25.182393 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.182186 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:33.182167528 +0000 UTC m=+18.302371837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:25.446349 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:25.446086 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:25.446784 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:25.446371 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:26.445431 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:26.445398 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:26.445620 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:26.445533 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:26.445865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:26.445849 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:26.445956 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:26.445939 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:27.445485 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:27.445450 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:27.445961 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:27.445572 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:28.445097 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:28.445065 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:28.445097 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:28.445091 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:28.445343 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:28.445214 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:28.445406 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:28.445371 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:28.494604 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:28.494032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" event={"ID":"4efe1724-f3ab-417d-bfdc-98ce7f4019e8","Type":"ContainerStarted","Data":"5168fa82231a1fc9625507df296e232620d5c7d96e29e29d7bea6a644054ff8b"} Jul 07 17:22:28.496028 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:28.495924 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" event={"ID":"ad89d569afe56b30feba4f95595fae64","Type":"ContainerStarted","Data":"f863f4149440a557da1514a0b0ea38b44ad401cd608a50f6bf6fcffdba1004ba"} Jul 07 17:22:28.515911 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:28.515852 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4n2ft" podStartSLOduration=3.357794963 podStartE2EDuration="13.515834889s" totalImagesPullingTime="10.158039926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:28.514507148 +0000 UTC m=+13.634711482" watchObservedRunningTime="2026-07-07 17:22:28.515834889 +0000 UTC m=+13.636039220" Jul 07 17:22:29.445380 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.445135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:29.445523 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:29.445497 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:29.498762 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.498725 2569 generic.go:356] "Generic (PLEG): container finished" podID="e30e6115e2341600f25267a786c8ff77" containerID="3c874bc679c8d660293fe5d8768984dd0917e7081e30f3a962ee49630b781fb8" exitCode=0 Jul 07 17:22:29.499258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.498817 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" event={"ID":"e30e6115e2341600f25267a786c8ff77","Type":"ContainerDied","Data":"3c874bc679c8d660293fe5d8768984dd0917e7081e30f3a962ee49630b781fb8"} Jul 07 17:22:29.500270 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.500247 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="e002456a6e2069d372ce5350d37cbe061d869a8b53859df173f70af07938c35b" exitCode=0 Jul 07 17:22:29.500369 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.500322 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"e002456a6e2069d372ce5350d37cbe061d869a8b53859df173f70af07938c35b"} Jul 07 17:22:29.501868 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.501829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pbt7g" event={"ID":"66ff5915-cd61-45ea-8199-1b41e636c35f","Type":"ContainerStarted","Data":"128b7e50b7d5fb5ab4d75dde71be8c2292c029c48e0fe704469db8e26f394a2b"} Jul 07 17:22:29.503360 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.503302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkd9s" event={"ID":"595c820e-0463-4011-851d-f3dea5f189e6","Type":"ContainerStarted","Data":"403c02335baf7e9f223914adc8b4a6926495a5344f964bc5af2ca67a4f3178cb"} Jul 07 17:22:29.504772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.504752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" event={"ID":"dde65482-9be3-4c80-8e60-d9900979c28b","Type":"ContainerStarted","Data":"f3f188ffb12f7c79291c170750d3372f053ee0d5419abf4e34ebf4b3c3f1275c"} Jul 07 17:22:29.506175 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.506149 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6whmp" event={"ID":"8addf0d9-b1e2-482f-a378-778455447ccf","Type":"ContainerStarted","Data":"9dedf5ce0b2998f084dac5c8fd1c40b44def0fb055d0b74cefc6f28e3446bede"} Jul 07 17:22:29.507786 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.507763 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnzsh" event={"ID":"75eaed74-372b-4f2d-a80e-bfb1b0398601","Type":"ContainerStarted","Data":"f808328b1871ce4bdd4c5ff04513c5cdaebb7d7d8d982b680cfdf2602c30a688"} Jul 07 17:22:29.521115 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.521080 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-197.ec2.internal" podStartSLOduration=13.521069693 podStartE2EDuration="13.521069693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:28.530574586 +0000 UTC m=+13.650778917" watchObservedRunningTime="2026-07-07 17:22:29.521069693 +0000 UTC m=+14.641274023" Jul 07 17:22:29.543805 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.543760 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6whmp" podStartSLOduration=4.528049819 podStartE2EDuration="14.543744339s" totalImagesPullingTime="10.01569452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:29.54372594 +0000 UTC m=+14.663930269" watchObservedRunningTime="2026-07-07 17:22:29.543744339 +0000 UTC m=+14.663948670" Jul 07 17:22:29.591771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.591730 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qkd9s" podStartSLOduration=4.439281487 podStartE2EDuration="14.591712404s" totalImagesPullingTime="10.152430917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:29.568671082 +0000 UTC m=+14.688875412" watchObservedRunningTime="2026-07-07 17:22:29.591712404 +0000 UTC m=+14.711916716" Jul 07 17:22:29.592389 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.592359 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pbt7g" podStartSLOduration=4.454316469 podStartE2EDuration="14.59234854s" totalImagesPullingTime="10.138032071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:29.591706849 +0000 UTC m=+14.711911181" watchObservedRunningTime="2026-07-07 17:22:29.59234854 +0000 UTC m=+14.712552871" Jul 07 17:22:29.610945 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:29.610898 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hnzsh" podStartSLOduration=4.407733229 podStartE2EDuration="14.610882933s" totalImagesPullingTime="10.203149704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:29.61033422 +0000 UTC m=+14.730538542" watchObservedRunningTime="2026-07-07 17:22:29.610882933 +0000 UTC m=+14.731087263" Jul 07 17:22:30.445078 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:30.445048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:30.445230 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:30.445147 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:30.445577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:30.445420 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:30.445577 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:30.445542 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:30.511589 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:30.511514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" event={"ID":"e30e6115e2341600f25267a786c8ff77","Type":"ContainerStarted","Data":"8611f55a3607cf7871252d6893ae117422b62f053a01fdba9b584dd1b8a9c669"} Jul 07 17:22:30.847953 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:30.847912 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 17:22:31.288834 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.288807 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:31.289444 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.289425 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:31.309697 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.309644 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-197.ec2.internal" podStartSLOduration=15.309629441 podStartE2EDuration="15.309629441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:30.538160306 +0000 UTC m=+15.658364635" watchObservedRunningTime="2026-07-07 17:22:31.309629441 +0000 UTC m=+16.429833770" Jul 07 17:22:31.365797 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.365659 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:22:30.847931072Z","UUID":"c507b5f5-5905-43f4-9977-99449f50e498","Handler":null,"Name":"","Endpoint":""} Jul 07 17:22:31.368121 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.367982 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 17:22:31.368121 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.368029 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 17:22:31.445072 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.445039 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:31.445311 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:31.445157 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:31.517378 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:31.517343 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" event={"ID":"dde65482-9be3-4c80-8e60-d9900979c28b","Type":"ContainerStarted","Data":"2f70b6a1c9eb969c8bdec69134bbb3276750e33bbc5b664b0a12852c85dd2b5f"} Jul 07 17:22:32.445357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:32.445328 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:32.445657 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:32.445460 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:32.445857 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:32.445832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:32.445986 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:32.445929 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:32.519745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:32.519711 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 17:22:33.038964 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:33.038738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:33.039137 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.038892 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:33.039137 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.039054 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.039033728 +0000 UTC m=+34.159238059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:33.140155 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:33.140122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:33.140367 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.140266 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:33.140367 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.140352 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.140336081 +0000 UTC m=+34.260540394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:33.241510 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:33.241467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:33.241712 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.241630 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:33.241712 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.241648 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:33.241712 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.241660 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:33.241874 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.241722 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.24170469 +0000 UTC m=+34.361909018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:33.445186 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:33.445066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:33.445371 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:33.445202 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:34.445748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:34.445709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:34.446266 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:34.445864 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:34.446266 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:34.445931 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:34.446266 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:34.446051 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:35.447316 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:35.445724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:35.447316 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:35.445866 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:36.445256 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:36.445219 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:36.445458 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:36.445297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:36.445458 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:36.445401 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:36.445573 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:36.445548 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:37.445694 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:37.445656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:37.446174 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:37.445787 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:38.445206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:38.445175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:38.445420 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:38.445177 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:38.445420 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:38.445344 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:38.445420 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:38.445406 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:39.445332 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:39.445293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:39.445722 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:39.445431 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:40.384933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:40.384900 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:40.385113 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:40.385099 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 17:22:40.385531 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:40.385504 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pbt7g" Jul 07 17:22:40.445672 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:40.445632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:40.446093 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:40.445759 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:40.446093 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:40.445820 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:40.446093 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:40.445945 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:41.445560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:41.445525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:41.445732 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:41.445664 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:42.444881 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.444847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:42.445025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.444857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:42.445083 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:42.445013 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:42.445136 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:42.445077 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:42.546301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.546039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b2qt" event={"ID":"e009614e-0bb8-4458-9c73-33a9268f90f3","Type":"ContainerStarted","Data":"a6fe25a2098c667e4de0807e35fe1988f64ab10f8328ea47eb3382498d8577a1"} Jul 07 17:22:42.548515 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.548487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"a0bf721fee796f865ef473cabb4e64d265baefcb6d17a360fa6869e5a954c84c"} Jul 07 17:22:42.548627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.548570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"c623b41ba6a3bcc060ec21ad01dc9dd60a0b556db6d94ceaf0e65e16f364220e"} Jul 07 17:22:42.550617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.550588 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerStarted","Data":"62263dccfd66fa7c2a175f7dce8a3a69e77159d7631f711bdae95a388fe9502b"} Jul 07 17:22:42.553051 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.552955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" event={"ID":"dde65482-9be3-4c80-8e60-d9900979c28b","Type":"ContainerStarted","Data":"21392df6c2c0eecbefaf9196c2fd863c0464cef0b87fff0ed1ffed699163c5fa"} Jul 07 17:22:42.596158 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.596116 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9vnw" podStartSLOduration=5.625947216 podStartE2EDuration="27.596101095s" totalImagesPullingTime="21.970153879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:42.595655447 +0000 UTC m=+27.715859774" watchObservedRunningTime="2026-07-07 17:22:42.596101095 +0000 UTC m=+27.716305424" Jul 07 17:22:42.596379 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:42.596358 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5b2qt" podStartSLOduration=2.963505353 podStartE2EDuration="27.596352292s" totalImagesPullingTime="24.632846939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:42.569956174 +0000 UTC m=+27.690160505" watchObservedRunningTime="2026-07-07 17:22:42.596352292 +0000 UTC m=+27.716556625" Jul 07 17:22:43.445745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.445720 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:43.445904 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:43.445818 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:43.558157 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.558123 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="62263dccfd66fa7c2a175f7dce8a3a69e77159d7631f711bdae95a388fe9502b" exitCode=0 Jul 07 17:22:43.558773 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.558179 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"62263dccfd66fa7c2a175f7dce8a3a69e77159d7631f711bdae95a388fe9502b"} Jul 07 17:22:43.561483 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.561446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"c1ca832e4de27bf773e183f1c3dec17bf70a2ffde72bac37966b5c722bc06189"} Jul 07 17:22:43.561603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.561532 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"1a067d0b0cf847b13a0e54205d8109ff0ed8a2a4fb18671a27e9d85a93dbe059"} Jul 07 17:22:43.561603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.561570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"a89f5d81aac278fd4b35e50502ac9ee99f325a5b861e79e263ae575c23b4f3b1"} Jul 07 17:22:43.561603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:43.561597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"2fd441d1f5c79b04a29061e4e7d66c4c01e6ef147e34742b01c45d5bceaead38"} Jul 07 17:22:44.445614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:44.445584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:44.445614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:44.445607 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:44.445779 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:44.445707 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:44.445840 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:44.445822 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:44.565803 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:44.565705 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="b642ac8528b685e5a737e46a1a6372ab5c48b686e02eee9df34528d966b1cfda" exitCode=0 Jul 07 17:22:44.565803 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:44.565759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"b642ac8528b685e5a737e46a1a6372ab5c48b686e02eee9df34528d966b1cfda"} Jul 07 17:22:45.446221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:45.446190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:45.446430 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:45.446276 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:45.570858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:45.570824 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"0d48706a94b8231f9d553f7a29e9d47bb81132a2886e7f3def9ae1882727fd35"} Jul 07 17:22:46.445837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:46.445812 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:46.445998 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:46.445819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:46.445998 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:46.445912 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:46.446074 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:46.445994 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:46.575233 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:46.575193 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="1074b8c32665345d855835f4bdbfd396677c509e1fecb8b6e0a192a87f8ee63e" exitCode=0 Jul 07 17:22:46.575696 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:46.575252 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"1074b8c32665345d855835f4bdbfd396677c509e1fecb8b6e0a192a87f8ee63e"} Jul 07 17:22:47.446088 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.445921 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:47.446254 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:47.446167 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:47.581345 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.581301 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" event={"ID":"1739a7b0-6920-4604-aa9d-939f5c92e46b","Type":"ContainerStarted","Data":"970678751eaf3c304a9770e6f072717743a1c3aa43cda4f58fd5798d9d4d45c4"} Jul 07 17:22:47.581765 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.581603 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:47.581765 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.581655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:47.598752 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.598729 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:47.619723 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:47.619679 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" podStartSLOduration=7.919797626 podStartE2EDuration="32.619663806s" totalImagesPullingTime="24.69986618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:47.619357001 +0000 UTC m=+32.739561333" watchObservedRunningTime="2026-07-07 17:22:47.619663806 +0000 UTC m=+32.739868180" Jul 07 17:22:48.445637 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:48.445594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:48.445790 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:48.445640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:48.445790 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:48.445738 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:48.445916 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:48.445894 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:48.594306 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:48.585056 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:48.611405 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:48.611380 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:22:49.065237 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.065207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:49.065456 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.065359 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:49.065456 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.065413 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret podName:87d5a160-779e-4370-b6fb-96a892b2dbca nodeName:}" failed. No retries permitted until 2026-07-07 17:23:21.065400481 +0000 UTC m=+66.185604789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret") pod "global-pull-secret-syncer-h8x4x" (UID: "87d5a160-779e-4370-b6fb-96a892b2dbca") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:49.165650 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.165618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:49.165820 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.165737 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:49.165820 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.165797 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs podName:ca3aed3c-1f0d-466a-834e-86aa02f6e884 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:21.165783095 +0000 UTC m=+66.285987406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs") pod "network-metrics-daemon-9sfzm" (UID: "ca3aed3c-1f0d-466a-834e-86aa02f6e884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:49.266607 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.266574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:49.266742 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.266713 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:49.266742 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.266728 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:49.266742 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.266736 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2gp8q for pod openshift-network-diagnostics/network-check-target-88r7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:49.266856 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.266781 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q podName:20090afb-e3c1-429c-8a1c-bed594800579 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:21.26677005 +0000 UTC m=+66.386974358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gp8q" (UniqueName: "kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q") pod "network-check-target-88r7j" (UID: "20090afb-e3c1-429c-8a1c-bed594800579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:49.445589 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.445247 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:49.445589 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.445389 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:49.459718 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.459687 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88r7j"] Jul 07 17:22:49.460324 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.460302 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8x4x"] Jul 07 17:22:49.460496 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.460401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:49.460564 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.460503 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:49.462216 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.462197 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9sfzm"] Jul 07 17:22:49.462334 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.462320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:49.462445 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.462428 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:49.586171 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:49.586136 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:49.586336 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:49.586252 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:51.445758 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:51.445813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:51.445835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:51.445914 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:51.446045 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:51.446310 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:51.446148 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:53.445500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:53.445463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:53.445948 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:53.445477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:53.445948 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:53.445568 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x4x" podUID="87d5a160-779e-4370-b6fb-96a892b2dbca" Jul 07 17:22:53.445948 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:53.445651 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sfzm" podUID="ca3aed3c-1f0d-466a-834e-86aa02f6e884" Jul 07 17:22:53.445948 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:53.445478 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:53.445948 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:53.445720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88r7j" podUID="20090afb-e3c1-429c-8a1c-bed594800579" Jul 07 17:22:53.596720 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:53.596635 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="d0dd101f588e8c1687b228de08cfb1b8d2e6ac6dc3049338a666fba8aaa3cffe" exitCode=0 Jul 07 17:22:53.596720 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:53.596704 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"d0dd101f588e8c1687b228de08cfb1b8d2e6ac6dc3049338a666fba8aaa3cffe"} Jul 07 17:22:54.600925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:54.600896 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e87ea91-61dd-48f2-a238-5bc98e293851" containerID="4bdecc0966049c5b7bc4b2fceec807d5fcac3866fe0262b60e910a89badc95ec" exitCode=0 Jul 07 17:22:54.601504 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:54.600962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerDied","Data":"4bdecc0966049c5b7bc4b2fceec807d5fcac3866fe0262b60e910a89badc95ec"} Jul 07 17:22:55.114908 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.114881 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-197.ec2.internal" event="NodeReady" Jul 07 17:22:55.115045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.114989 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 17:22:55.171108 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.170802 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:22:55.186016 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.185996 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-85ggm"] Jul 07 17:22:55.189145 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.186382 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.189824 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.189802 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 17:22:55.190026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.190008 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 17:22:55.190088 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.190053 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5jqz\"" Jul 07 17:22:55.190347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.190327 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 17:22:55.198821 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.198798 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz"] Jul 07 17:22:55.199278 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.199261 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.199733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.199709 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 17:22:55.202394 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.202378 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 17:22:55.202621 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.202607 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.202694 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.202638 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt4pt\"" Jul 07 17:22:55.202865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.202847 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.206594 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.206575 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 17:22:55.211521 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.211494 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx"] Jul 07 17:22:55.211664 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.211648 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.212401 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.212380 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 17:22:55.215729 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.215712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 17:22:55.215970 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.215951 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.215970 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.215961 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 17:22:55.216149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.216091 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7lgkq\"" Jul 07 17:22:55.216149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.216118 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.226761 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.226743 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-784748979c-znp9d"] Jul 07 17:22:55.226873 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.226859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" Jul 07 17:22:55.230534 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.230513 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cbtcp\"" Jul 07 17:22:55.230888 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.230868 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.231045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.231023 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.242061 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.242039 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856"] Jul 07 17:22:55.242158 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.242145 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.245176 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w6x9j\"" Jul 07 17:22:55.245176 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245175 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.245359 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 17:22:55.245418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245161 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 17:22:55.245470 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245434 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 17:22:55.245470 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245436 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.245564 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.245512 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 17:22:55.255453 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255432 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-85ggm"] Jul 07 17:22:55.255539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz"] Jul 07 17:22:55.255539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255477 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx"] Jul 07 17:22:55.255539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856"] Jul 07 17:22:55.255539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255525 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dxjt7"] Jul 07 17:22:55.255726 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.255582 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.258164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.258145 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 17:22:55.258247 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.258212 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.258454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.258437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.258531 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.258455 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hh786\"" Jul 07 17:22:55.266538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.266520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:22:55.266622 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.266549 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2ffz"] Jul 07 17:22:55.266660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.266651 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.269438 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.269423 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 17:22:55.269497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.269439 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 17:22:55.270106 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.270092 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pgv6p\"" Jul 07 17:22:55.277257 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.277239 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-784748979c-znp9d"] Jul 07 17:22:55.277336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.277267 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8"] Jul 07 17:22:55.277369 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.277357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.280521 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.280504 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 17:22:55.280625 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.280524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h4cdr\"" Jul 07 17:22:55.280625 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.280541 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 17:22:55.280625 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.280549 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.281023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.280990 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.286767 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.286752 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 17:22:55.294945 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.294929 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp"] Jul 07 17:22:55.295049 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.295031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" Jul 07 17:22:55.297849 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.297830 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.297928 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.297920 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.298065 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.298049 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-54jkc\"" Jul 07 17:22:55.308684 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-snapshots\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.308782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.308782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.308782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.308782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph9s\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.308933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.308933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pld89\" (UniqueName: \"kubernetes.io/projected/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-kube-api-access-pld89\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.308933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.308933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-serving-cert\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.308933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2sv\" (UniqueName: \"kubernetes.io/projected/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-kube-api-access-hk2sv\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-tmp\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.308991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.309022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.309039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.309079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.309118 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.309114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.313206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.313127 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bqgft"] Jul 07 17:22:55.313307 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.313194 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.315952 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.315932 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.316024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.316001 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mksm8\"" Jul 07 17:22:55.316094 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.316004 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 17:22:55.316312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.316296 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.316496 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.316485 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 17:22:55.328499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.328482 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt"] Jul 07 17:22:55.328600 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.328504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.330913 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.330897 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 17:22:55.330991 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.330903 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pcwpt\"" Jul 07 17:22:55.331672 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.331658 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.331975 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.331960 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.338929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.338913 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxjt7"] Jul 07 17:22:55.339018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.338944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp"] Jul 07 17:22:55.339018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.338958 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8"] Jul 07 17:22:55.339018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.338970 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2ffz"] Jul 07 17:22:55.339018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.338992 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqgft"] Jul 07 17:22:55.339018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.339003 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt"] Jul 07 17:22:55.339263 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.339029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.341988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.341970 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nt4bs\"" Jul 07 17:22:55.342066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.341998 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:55.342066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.341974 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:55.342066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.341974 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 17:22:55.342066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.342009 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 17:22:55.409396 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.409396 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph9s\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.409617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.409617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409438 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pld89\" (UniqueName: \"kubernetes.io/projected/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-kube-api-access-pld89\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.409617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.409617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdsfx\" (UniqueName: \"kubernetes.io/projected/6b464785-40b5-49ff-a829-008160bc0045-kube-api-access-hdsfx\") pod \"network-check-source-989f8b8dc-2c8q8\" (UID: \"6b464785-40b5-49ff-a829-008160bc0045\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" Jul 07 17:22:55.409772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-config\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.409772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5876c8dd-c233-475f-a608-85c8f137c6c7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.409772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5876c8dd-c233-475f-a608-85c8f137c6c7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.409772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-tmp-dir\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.409900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409809 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.409900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p2g\" (UniqueName: \"kubernetes.io/projected/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-kube-api-access-v4p2g\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.409991 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.409991 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.409959 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.410090 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-snapshots\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410090 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.410090 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.410302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dndr\" (UniqueName: \"kubernetes.io/projected/2807ef90-5a4a-4bc3-ac29-240205e2890b-kube-api-access-9dndr\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.410302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.410302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-trusted-ca\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.410302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410507 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7d6\" (UniqueName: \"kubernetes.io/projected/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-kube-api-access-5n7d6\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.410560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410435 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.410560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.410660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-serving-cert\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqvd\" (UniqueName: \"kubernetes.io/projected/4d370278-be42-4bb8-a24e-1e035a961a3d-kube-api-access-tvqvd\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgqfx\" (UniqueName: \"kubernetes.io/projected/5876c8dd-c233-475f-a608-85c8f137c6c7-kube-api-access-kgqfx\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410693 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410714 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-tmp\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d370278-be42-4bb8-a24e-1e035a961a3d-serving-cert\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410755 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.410809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410713 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-snapshots\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2sv\" (UniqueName: \"kubernetes.io/projected/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-kube-api-access-hk2sv\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-default-certificate\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxpk\" (UniqueName: \"kubernetes.io/projected/13577d5c-0545-4095-8369-48b4d6557a61-kube-api-access-8jxpk\") pod \"volume-data-source-validator-5d456fdbcd-ddhbx\" (UID: \"13577d5c-0545-4095-8369-48b4d6557a61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-stats-auth\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410966 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-tmp\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.410985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.411060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.411030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.411092 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.411105 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.411151 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:55.911134203 +0000 UTC m=+41.031338529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.411322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-config-volume\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.411353 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.411359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.411486 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.411450 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:55.411834 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.411503 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:55.911482468 +0000 UTC m=+41.031686777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:55.411834 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.411611 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.414347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.414316 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-serving-cert\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.414347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.414334 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.414518 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.414357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.421859 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.421836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2sv\" (UniqueName: \"kubernetes.io/projected/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-kube-api-access-hk2sv\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.422121 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.422102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pld89\" (UniqueName: \"kubernetes.io/projected/79a1fb9f-b7f2-4692-9b45-c9871bc7eb58-kube-api-access-pld89\") pod \"insights-operator-5c847dcfc4-85ggm\" (UID: \"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58\") " pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.422364 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.422348 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph9s\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.422972 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.422957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.444913 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.444892 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:22:55.444913 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.444894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:22:55.445068 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.444903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:22:55.447821 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.447801 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmhtn\"" Jul 07 17:22:55.447932 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.447866 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7rcfs\"" Jul 07 17:22:55.448067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.448049 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:22:55.448134 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.448053 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:22:55.507913 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.507883 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" Jul 07 17:22:55.512185 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512160 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdsfx\" (UniqueName: \"kubernetes.io/projected/6b464785-40b5-49ff-a829-008160bc0045-kube-api-access-hdsfx\") pod \"network-check-source-989f8b8dc-2c8q8\" (UID: \"6b464785-40b5-49ff-a829-008160bc0045\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-config\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5876c8dd-c233-475f-a608-85c8f137c6c7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5876c8dd-c233-475f-a608-85c8f137c6c7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.512317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-tmp-dir\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.512528 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512341 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:22:55.512528 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512414 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.012393273 +0000 UTC m=+41.132597601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:22:55.512528 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512340 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:22:55.512528 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512455 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls podName:2807ef90-5a4a-4bc3-ac29-240205e2890b nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.012446258 +0000 UTC m=+41.132650567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ld856" (UID: "2807ef90-5a4a-4bc3-ac29-240205e2890b") : secret "samples-operator-tls" not found Jul 07 17:22:55.512676 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512604 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p2g\" (UniqueName: \"kubernetes.io/projected/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-kube-api-access-v4p2g\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.512676 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.512676 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512664 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-tmp-dir\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.512812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dndr\" (UniqueName: \"kubernetes.io/projected/2807ef90-5a4a-4bc3-ac29-240205e2890b-kube-api-access-9dndr\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.512812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9d8bc1-de71-42e4-becc-217e80ac8444-config\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.512812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512746 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-trusted-ca\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.512812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7d6\" (UniqueName: \"kubernetes.io/projected/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-kube-api-access-5n7d6\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsdt\" (UniqueName: \"kubernetes.io/projected/5391d841-23bf-4e93-848e-b2788bc05d76-kube-api-access-vqsdt\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512749 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwvl\" (UniqueName: \"kubernetes.io/projected/cc9d8bc1-de71-42e4-becc-217e80ac8444-kube-api-access-5nwvl\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512888 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.012868664 +0000 UTC m=+41.133072980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqvd\" (UniqueName: \"kubernetes.io/projected/4d370278-be42-4bb8-a24e-1e035a961a3d-kube-api-access-tvqvd\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.512970 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.012955423 +0000 UTC m=+41.133159752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.512989 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5876c8dd-c233-475f-a608-85c8f137c6c7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgqfx\" (UniqueName: \"kubernetes.io/projected/5876c8dd-c233-475f-a608-85c8f137c6c7-kube-api-access-kgqfx\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d370278-be42-4bb8-a24e-1e035a961a3d-serving-cert\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513068 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-config\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-default-certificate\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5391d841-23bf-4e93-848e-b2788bc05d76-cert\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.513159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxpk\" (UniqueName: \"kubernetes.io/projected/13577d5c-0545-4095-8369-48b4d6557a61-kube-api-access-8jxpk\") pod \"volume-data-source-validator-5d456fdbcd-ddhbx\" (UID: \"13577d5c-0545-4095-8369-48b4d6557a61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" Jul 07 17:22:55.513926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-stats-auth\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.513926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-config-volume\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.513926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9d8bc1-de71-42e4-becc-217e80ac8444-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.513926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513529 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d370278-be42-4bb8-a24e-1e035a961a3d-trusted-ca\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.513926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.513747 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-config-volume\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.514900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.514878 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5876c8dd-c233-475f-a608-85c8f137c6c7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.515578 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.515560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-stats-auth\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.516028 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.515916 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d370278-be42-4bb8-a24e-1e035a961a3d-serving-cert\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.516028 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.515921 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-default-certificate\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.524728 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.524707 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7d6\" (UniqueName: \"kubernetes.io/projected/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-kube-api-access-5n7d6\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:55.524974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.524943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqvd\" (UniqueName: \"kubernetes.io/projected/4d370278-be42-4bb8-a24e-1e035a961a3d-kube-api-access-tvqvd\") pod \"console-operator-68b757fb67-p2ffz\" (UID: \"4d370278-be42-4bb8-a24e-1e035a961a3d\") " pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.524974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.524953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dndr\" (UniqueName: \"kubernetes.io/projected/2807ef90-5a4a-4bc3-ac29-240205e2890b-kube-api-access-9dndr\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:55.525178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.525022 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdsfx\" (UniqueName: \"kubernetes.io/projected/6b464785-40b5-49ff-a829-008160bc0045-kube-api-access-hdsfx\") pod \"network-check-source-989f8b8dc-2c8q8\" (UID: \"6b464785-40b5-49ff-a829-008160bc0045\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" Jul 07 17:22:55.525178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.525134 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgqfx\" (UniqueName: \"kubernetes.io/projected/5876c8dd-c233-475f-a608-85c8f137c6c7-kube-api-access-kgqfx\") pod \"kube-storage-version-migrator-operator-846bf668f7-jwwhp\" (UID: \"5876c8dd-c233-475f-a608-85c8f137c6c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.525596 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.525580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxpk\" (UniqueName: \"kubernetes.io/projected/13577d5c-0545-4095-8369-48b4d6557a61-kube-api-access-8jxpk\") pod \"volume-data-source-validator-5d456fdbcd-ddhbx\" (UID: \"13577d5c-0545-4095-8369-48b4d6557a61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" Jul 07 17:22:55.525674 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.525660 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p2g\" (UniqueName: \"kubernetes.io/projected/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-kube-api-access-v4p2g\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:55.536909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.536887 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" Jul 07 17:22:55.586615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.586436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:22:55.602897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.602676 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" Jul 07 17:22:55.608651 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.608622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" event={"ID":"2e87ea91-61dd-48f2-a238-5bc98e293851","Type":"ContainerStarted","Data":"90f51e9b863330fa6dda8d061899e9f71d760672b61a597c3edddcd65ffe010b"} Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.614073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwvl\" (UniqueName: \"kubernetes.io/projected/cc9d8bc1-de71-42e4-becc-217e80ac8444-kube-api-access-5nwvl\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.614157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5391d841-23bf-4e93-848e-b2788bc05d76-cert\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.614208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9d8bc1-de71-42e4-becc-217e80ac8444-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.614358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9d8bc1-de71-42e4-becc-217e80ac8444-config\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.614393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsdt\" (UniqueName: \"kubernetes.io/projected/5391d841-23bf-4e93-848e-b2788bc05d76-kube-api-access-vqsdt\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.616628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.616582 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc9d8bc1-de71-42e4-becc-217e80ac8444-config\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.620711 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.620674 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" Jul 07 17:22:55.626766 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.626712 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5391d841-23bf-4e93-848e-b2788bc05d76-cert\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.627110 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.627062 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9d8bc1-de71-42e4-becc-217e80ac8444-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.634006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.631462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwvl\" (UniqueName: \"kubernetes.io/projected/cc9d8bc1-de71-42e4-becc-217e80ac8444-kube-api-access-5nwvl\") pod \"service-ca-operator-5d46d9ffd4-dk2rt\" (UID: \"cc9d8bc1-de71-42e4-becc-217e80ac8444\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.634006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.632026 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsdt\" (UniqueName: \"kubernetes.io/projected/5391d841-23bf-4e93-848e-b2788bc05d76-kube-api-access-vqsdt\") pod \"ingress-canary-bqgft\" (UID: \"5391d841-23bf-4e93-848e-b2788bc05d76\") " pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.637149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.636073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqgft" Jul 07 17:22:55.645631 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.644118 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pbqfz" podStartSLOduration=8.68219413 podStartE2EDuration="40.644097108s" totalImagesPullingTime="31.961902978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:55.64246807 +0000 UTC m=+40.762672401" watchObservedRunningTime="2026-07-07 17:22:55.644097108 +0000 UTC m=+40.764301439" Jul 07 17:22:55.646716 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.646699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" Jul 07 17:22:55.768250 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.764722 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx"] Jul 07 17:22:55.768250 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.768240 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-85ggm"] Jul 07 17:22:55.780188 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.777046 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13577d5c_0545_4095_8369_48b4d6557a61.slice/crio-ccaa39a21046dded67839bb35060d11fcccc004ea83994b8229cf4abf387c261 WatchSource:0}: Error finding container ccaa39a21046dded67839bb35060d11fcccc004ea83994b8229cf4abf387c261: Status 404 returned error can't find the container with id ccaa39a21046dded67839bb35060d11fcccc004ea83994b8229cf4abf387c261 Jul 07 17:22:55.785243 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.784044 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a1fb9f_b7f2_4692_9b45_c9871bc7eb58.slice/crio-baad5e708b61b3252fce41532ced2a34c7127ec8c13d372022f0835334bdf6e6 WatchSource:0}: Error finding container baad5e708b61b3252fce41532ced2a34c7127ec8c13d372022f0835334bdf6e6: Status 404 returned error can't find the container with id baad5e708b61b3252fce41532ced2a34c7127ec8c13d372022f0835334bdf6e6 Jul 07 17:22:55.813895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.813787 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:55.851525 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.851362 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2ffz"] Jul 07 17:22:55.855143 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.855107 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d370278_be42_4bb8_a24e_1e035a961a3d.slice/crio-e0769113ee095dc8fed7ec93e8e383b417f15b2152e825420dedcd3abd7f33bb WatchSource:0}: Error finding container e0769113ee095dc8fed7ec93e8e383b417f15b2152e825420dedcd3abd7f33bb: Status 404 returned error can't find the container with id e0769113ee095dc8fed7ec93e8e383b417f15b2152e825420dedcd3abd7f33bb Jul 07 17:22:55.859065 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.858911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8"] Jul 07 17:22:55.863009 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.862982 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b464785_40b5_49ff_a829_008160bc0045.slice/crio-7d7eb9c6871cee13191782b64106c8ba59cb83ee50483a12ecbe48b76db93324 WatchSource:0}: Error finding container 7d7eb9c6871cee13191782b64106c8ba59cb83ee50483a12ecbe48b76db93324: Status 404 returned error can't find the container with id 7d7eb9c6871cee13191782b64106c8ba59cb83ee50483a12ecbe48b76db93324 Jul 07 17:22:55.873874 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.873848 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp"] Jul 07 17:22:55.876437 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.876416 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5876c8dd_c233_475f_a608_85c8f137c6c7.slice/crio-69c2b40c6fe4a44ee261499ae39fefec5fc24967ecd335bb97a6cb0f5045051a WatchSource:0}: Error finding container 69c2b40c6fe4a44ee261499ae39fefec5fc24967ecd335bb97a6cb0f5045051a: Status 404 returned error can't find the container with id 69c2b40c6fe4a44ee261499ae39fefec5fc24967ecd335bb97a6cb0f5045051a Jul 07 17:22:55.884707 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.884677 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqgft"] Jul 07 17:22:55.889677 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.889657 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5391d841_23bf_4e93_848e_b2788bc05d76.slice/crio-1dcd8e1d11aca433def82702996fbb5f850edbe0553d0c477f0811debf7b88b3 WatchSource:0}: Error finding container 1dcd8e1d11aca433def82702996fbb5f850edbe0553d0c477f0811debf7b88b3: Status 404 returned error can't find the container with id 1dcd8e1d11aca433def82702996fbb5f850edbe0553d0c477f0811debf7b88b3 Jul 07 17:22:55.902712 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.902689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt"] Jul 07 17:22:55.905294 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:22:55.905257 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9d8bc1_de71_42e4_becc_217e80ac8444.slice/crio-870d0e758d2d21c1c1193c73f760a7288da8960c6ceb67a1069db54b445c28b9 WatchSource:0}: Error finding container 870d0e758d2d21c1c1193c73f760a7288da8960c6ceb67a1069db54b445c28b9: Status 404 returned error can't find the container with id 870d0e758d2d21c1c1193c73f760a7288da8960c6ceb67a1069db54b445c28b9 Jul 07 17:22:55.917848 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.917802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:55.917848 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.917835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:55.917954 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.917937 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:22:55.917954 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.917951 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:22:55.918013 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.917988 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.91797538 +0000 UTC m=+42.038179724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:22:55.918053 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.918014 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:55.918053 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:55.918051 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:56.918040568 +0000 UTC m=+42.038244879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:55.925835 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:55.925812 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:56.018636 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.018607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:56.018733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.018643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:56.018733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.018674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:56.018733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.018702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:56.018835 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018754 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:22:56.018835 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018764 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:22:56.018835 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018805 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:57.018790608 +0000 UTC m=+42.138994915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:22:56.018835 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018806 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:22:56.018835 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018821 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls podName:2807ef90-5a4a-4bc3-ac29-240205e2890b nodeName:}" failed. No retries permitted until 2026-07-07 17:22:57.018814961 +0000 UTC m=+42.139019270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ld856" (UID: "2807ef90-5a4a-4bc3-ac29-240205e2890b") : secret "samples-operator-tls" not found Jul 07 17:22:56.018985 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018855 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:57.018840686 +0000 UTC m=+42.139045011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:22:56.018985 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.018875 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:57.018862219 +0000 UTC m=+42.139066550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:22:56.613752 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.613711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqgft" event={"ID":"5391d841-23bf-4e93-848e-b2788bc05d76","Type":"ContainerStarted","Data":"1dcd8e1d11aca433def82702996fbb5f850edbe0553d0c477f0811debf7b88b3"} Jul 07 17:22:56.615178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.615143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" event={"ID":"5876c8dd-c233-475f-a608-85c8f137c6c7","Type":"ContainerStarted","Data":"69c2b40c6fe4a44ee261499ae39fefec5fc24967ecd335bb97a6cb0f5045051a"} Jul 07 17:22:56.616477 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.616452 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" event={"ID":"6b464785-40b5-49ff-a829-008160bc0045","Type":"ContainerStarted","Data":"7d7eb9c6871cee13191782b64106c8ba59cb83ee50483a12ecbe48b76db93324"} Jul 07 17:22:56.617733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.617707 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" event={"ID":"4d370278-be42-4bb8-a24e-1e035a961a3d","Type":"ContainerStarted","Data":"e0769113ee095dc8fed7ec93e8e383b417f15b2152e825420dedcd3abd7f33bb"} Jul 07 17:22:56.618996 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.618972 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" event={"ID":"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58","Type":"ContainerStarted","Data":"baad5e708b61b3252fce41532ced2a34c7127ec8c13d372022f0835334bdf6e6"} Jul 07 17:22:56.620545 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.620520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" event={"ID":"cc9d8bc1-de71-42e4-becc-217e80ac8444","Type":"ContainerStarted","Data":"870d0e758d2d21c1c1193c73f760a7288da8960c6ceb67a1069db54b445c28b9"} Jul 07 17:22:56.621662 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.621638 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" event={"ID":"13577d5c-0545-4095-8369-48b4d6557a61","Type":"ContainerStarted","Data":"ccaa39a21046dded67839bb35060d11fcccc004ea83994b8229cf4abf387c261"} Jul 07 17:22:56.927557 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.927518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:56.927725 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:56.927575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:56.927797 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.927755 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:56.927855 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.927819 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:58.92779991 +0000 UTC m=+44.048004233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:56.928224 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.928202 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:22:56.928224 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.928222 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:22:56.928378 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:56.928271 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:58.928256267 +0000 UTC m=+44.048460577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.028808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.028854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.028905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.028953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029095 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:59.029076982 +0000 UTC m=+44.149281293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029397 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029446 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls podName:2807ef90-5a4a-4bc3-ac29-240205e2890b nodeName:}" failed. No retries permitted until 2026-07-07 17:22:59.029431389 +0000 UTC m=+44.149635700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ld856" (UID: "2807ef90-5a4a-4bc3-ac29-240205e2890b") : secret "samples-operator-tls" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029498 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029529 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:59.029519274 +0000 UTC m=+44.149723586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029575 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:22:57.029631 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.029604 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:59.029595305 +0000 UTC m=+44.149799628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:22:57.122234 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.121950 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp"] Jul 07 17:22:57.144043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.144007 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp"] Jul 07 17:22:57.144220 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.144122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.148575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.147860 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9xdwm\"" Jul 07 17:22:57.148575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.148079 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 17:22:57.148575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.148313 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 17:22:57.258249 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.258145 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:57.323912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.323852 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:57.333166 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.332878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.333166 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.333000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.393083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.392887 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:57.433905 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.433870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.434088 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.434051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.434241 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.434214 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:22:57.434405 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.434308 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:22:57.934269506 +0000 UTC m=+43.054473828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:22:57.435366 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.435262 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.497705 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.497654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:57.599365 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.599250 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:57.939589 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:57.939550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:57.940070 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.939698 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:22:57.940070 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:57.939759 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:22:58.939740029 +0000 UTC m=+44.059944354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:22:58.630204 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:58.630097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" event={"ID":"13577d5c-0545-4095-8369-48b4d6557a61","Type":"ContainerStarted","Data":"a81194b76f06643c075c8d025476aa6c61e77f7198a9d315665f23c509bd14de"} Jul 07 17:22:58.783444 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:58.783407 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:22:58.949805 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:58.949767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:58.949817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:58.949904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.949920 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.949939 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.949978 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.949986 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:02.949971769 +0000 UTC m=+48.070176080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.950024 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.950034 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:02.950021477 +0000 UTC m=+48.070225785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:22:58.950379 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:58.950073 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:00.950057545 +0000 UTC m=+46.070261871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:22:59.050780 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:59.050751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:59.050938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:59.050804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:22:59.050938 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.050908 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:22:59.050938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:59.050927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:22:59.051050 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.050959 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.050939685 +0000 UTC m=+48.171143996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:22:59.051050 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:22:59.050989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:22:59.051050 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.050993 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:22:59.051050 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.051044 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.051026525 +0000 UTC m=+48.171230846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:22:59.051050 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.051037 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:22:59.051247 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.051074 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.051062904 +0000 UTC m=+48.171267217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:22:59.051247 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:22:59.051086 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls podName:2807ef90-5a4a-4bc3-ac29-240205e2890b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.051079894 +0000 UTC m=+48.171284204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ld856" (UID: "2807ef90-5a4a-4bc3-ac29-240205e2890b") : secret "samples-operator-tls" not found Jul 07 17:23:00.535801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:00.535755 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:00.612790 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:00.612750 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:00.967071 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:00.967030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:00.967271 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:00.967230 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:23:00.967363 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:00.967330 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:04.96730491 +0000 UTC m=+50.087509222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:23:02.133953 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:02.133910 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:02.983941 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:02.983904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:02.984136 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:02.983950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:23:02.984136 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:02.984089 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:23:02.984136 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:02.984114 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:23:02.984330 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:02.984174 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:23:02.984330 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:02.984180 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:10.984157779 +0000 UTC m=+56.104362099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:23:02.984330 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:02.984260 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:10.984239647 +0000 UTC m=+56.104443969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:23:03.084605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:03.084569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:03.084605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:03.084610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:23:03.084836 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:03.084677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:03.084836 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084718 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:23:03.084836 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084776 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:11.084760744 +0000 UTC m=+56.204965052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:23:03.084836 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:03.084724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:03.084836 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084813 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:11.08479693 +0000 UTC m=+56.205001257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:23:03.085028 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084839 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:23:03.085028 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084850 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:23:03.085028 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084880 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:11.084869642 +0000 UTC m=+56.205073951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:23:03.085028 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:03.084899 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls podName:2807ef90-5a4a-4bc3-ac29-240205e2890b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:11.084886965 +0000 UTC m=+56.205091273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ld856" (UID: "2807ef90-5a4a-4bc3-ac29-240205e2890b") : secret "samples-operator-tls" not found Jul 07 17:23:04.149272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.149232 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:04.306629 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.306582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:04.466424 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.466334 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:04.650512 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.650440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqgft" event={"ID":"5391d841-23bf-4e93-848e-b2788bc05d76","Type":"ContainerStarted","Data":"21f9189124765475bf852aff1ce5aa0ff33ee0f029dbfaa5ea48c0bbb48f3b45"} Jul 07 17:23:04.652218 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.652173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" event={"ID":"5876c8dd-c233-475f-a608-85c8f137c6c7","Type":"ContainerStarted","Data":"7c5d9540d8dafa992e4e46898aec4d04d5ce7f551791e2f59b91d298a1b46f1a"} Jul 07 17:23:04.653926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.653901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" event={"ID":"6b464785-40b5-49ff-a829-008160bc0045","Type":"ContainerStarted","Data":"b40861ec728ae08419db6e549476eadac8d1d33a017bcd22975a5f6d935ca45c"} Jul 07 17:23:04.656077 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.656057 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/0.log" Jul 07 17:23:04.656149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.656091 2569 generic.go:356] "Generic (PLEG): container finished" podID="4d370278-be42-4bb8-a24e-1e035a961a3d" containerID="411f03b09715c01ccccd4f4c84238b4992510d5f3dc88d1a7552dc5961c1381b" exitCode=255 Jul 07 17:23:04.656339 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.656314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" event={"ID":"4d370278-be42-4bb8-a24e-1e035a961a3d","Type":"ContainerDied","Data":"411f03b09715c01ccccd4f4c84238b4992510d5f3dc88d1a7552dc5961c1381b"} Jul 07 17:23:04.656462 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.656419 2569 scope.go:117] "RemoveContainer" containerID="411f03b09715c01ccccd4f4c84238b4992510d5f3dc88d1a7552dc5961c1381b" Jul 07 17:23:04.666870 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.666554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" event={"ID":"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58","Type":"ContainerStarted","Data":"c19807e09ac97fc2631ed0d10a3ca29d520aaa70bc7e4f429f18b15f83085e9a"} Jul 07 17:23:04.668570 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.668549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" event={"ID":"cc9d8bc1-de71-42e4-becc-217e80ac8444","Type":"ContainerStarted","Data":"53571d3cf38d6f9891d227bcb6fe32f3e5d7114522c9a879244d0f57d521b280"} Jul 07 17:23:04.671126 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.671071 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ddhbx" podStartSLOduration=15.367261361 podStartE2EDuration="17.671053934s" totalImagesPullingTime="2.303792573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:58.652890001 +0000 UTC m=+43.773094321" watchObservedRunningTime="2026-07-07 17:23:04.671053934 +0000 UTC m=+49.791258254" Jul 07 17:23:04.671381 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.671345 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bqgft" podStartSLOduration=1.9866152430000001 podStartE2EDuration="9.671338052s" totalImagesPullingTime="7.684722809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.670185776 +0000 UTC m=+49.790390104" watchObservedRunningTime="2026-07-07 17:23:04.671338052 +0000 UTC m=+49.791542371" Jul 07 17:23:04.698026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.692242 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" podStartSLOduration=10.001135144 podStartE2EDuration="17.692226194s" totalImagesPullingTime="7.69109105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.691017144 +0000 UTC m=+49.811221477" watchObservedRunningTime="2026-07-07 17:23:04.692226194 +0000 UTC m=+49.812430514" Jul 07 17:23:04.723625 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.723536 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" podStartSLOduration=10.061382756 podStartE2EDuration="17.723514925s" totalImagesPullingTime="7.662132169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.722065425 +0000 UTC m=+49.842269756" watchObservedRunningTime="2026-07-07 17:23:04.723514925 +0000 UTC m=+49.843719258" Jul 07 17:23:04.770576 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.770528 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2c8q8" podStartSLOduration=5.066592573 podStartE2EDuration="12.770508073s" totalImagesPullingTime="7.7039155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.767571079 +0000 UTC m=+49.887775410" watchObservedRunningTime="2026-07-07 17:23:04.770508073 +0000 UTC m=+49.890712404" Jul 07 17:23:04.770930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:04.770898 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" podStartSLOduration=9.99197594 podStartE2EDuration="17.770890834s" totalImagesPullingTime="7.778914894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.746253305 +0000 UTC m=+49.866457637" watchObservedRunningTime="2026-07-07 17:23:04.770890834 +0000 UTC m=+49.891095165" Jul 07 17:23:05.005070 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.004972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:05.005224 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.005144 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:23:05.005224 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.005213 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:13.005197419 +0000 UTC m=+58.125401728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:23:05.047789 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.047746 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf"] Jul 07 17:23:05.051009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.050977 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.063016 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.062990 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf"] Jul 07 17:23:05.063144 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.063110 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" Jul 07 17:23:05.065922 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.065901 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6q7s4\"" Jul 07 17:23:05.066241 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.065932 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 17:23:05.066241 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.065963 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 17:23:05.106140 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.106109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk49q\" (UniqueName: \"kubernetes.io/projected/157a2dab-3b23-4255-8c68-2dc658d24263-kube-api-access-fk49q\") pod \"migrator-d57d58cc9-jlfsf\" (UID: \"157a2dab-3b23-4255-8c68-2dc658d24263\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" Jul 07 17:23:05.142052 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.142005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.207491 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.207460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk49q\" (UniqueName: \"kubernetes.io/projected/157a2dab-3b23-4255-8c68-2dc658d24263-kube-api-access-fk49q\") pod \"migrator-d57d58cc9-jlfsf\" (UID: \"157a2dab-3b23-4255-8c68-2dc658d24263\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" Jul 07 17:23:05.219377 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.219352 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk49q\" (UniqueName: \"kubernetes.io/projected/157a2dab-3b23-4255-8c68-2dc658d24263-kube-api-access-fk49q\") pod \"migrator-d57d58cc9-jlfsf\" (UID: \"157a2dab-3b23-4255-8c68-2dc658d24263\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" Jul 07 17:23:05.244541 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.244495 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.338964 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.338861 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.339102 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.338981 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5jl4g"] Jul 07 17:23:05.347904 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.347878 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.350781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.350637 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 17:23:05.350781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.350737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2gj6\"" Jul 07 17:23:05.350781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.350775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 17:23:05.354997 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.354978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jl4g"] Jul 07 17:23:05.374167 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.374142 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" Jul 07 17:23:05.410046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.410019 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwf5\" (UniqueName: \"kubernetes.io/projected/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-api-access-cmwf5\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.410213 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.410069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.410213 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.410168 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e3a6b56-8f0f-4368-9afc-d3d485c54416-data-volume\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.410330 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.410217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e3a6b56-8f0f-4368-9afc-d3d485c54416-crio-socket\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.410364 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.410342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.444658 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.444620 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.508831 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.508799 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:05.510860 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.510840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e3a6b56-8f0f-4368-9afc-d3d485c54416-crio-socket\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.510930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.510915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.511051 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.511036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwf5\" (UniqueName: \"kubernetes.io/projected/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-api-access-cmwf5\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.511099 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.511079 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:23:05.511099 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.511085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.511182 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.511040 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e3a6b56-8f0f-4368-9afc-d3d485c54416-crio-socket\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.511182 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.511119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e3a6b56-8f0f-4368-9afc-d3d485c54416-data-volume\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.511182 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.511136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:06.011119992 +0000 UTC m=+51.131324320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : secret "insights-runtime-extractor-tls" not found Jul 07 17:23:05.511337 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.511255 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:23:05.511388 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.511335 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:06.011317366 +0000 UTC m=+51.131521688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : configmap "kube-rbac-proxy" not found Jul 07 17:23:05.511494 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.511479 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e3a6b56-8f0f-4368-9afc-d3d485c54416-data-volume\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.525592 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.524730 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf"] Jul 07 17:23:05.526008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.525981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwf5\" (UniqueName: \"kubernetes.io/projected/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-api-access-cmwf5\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:05.529500 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:05.529455 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157a2dab_3b23_4255_8c68_2dc658d24263.slice/crio-cfc91b2edd57c00c5ecd196a625aa8e135182ac78c4a5bc58a9833037e51dfea WatchSource:0}: Error finding container cfc91b2edd57c00c5ecd196a625aa8e135182ac78c4a5bc58a9833037e51dfea: Status 404 returned error can't find the container with id cfc91b2edd57c00c5ecd196a625aa8e135182ac78c4a5bc58a9833037e51dfea Jul 07 17:23:05.586756 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.586717 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:05.586877 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.586770 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:05.673897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.673864 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:23:05.674260 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.674245 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/0.log" Jul 07 17:23:05.674323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.674306 2569 generic.go:356] "Generic (PLEG): container finished" podID="4d370278-be42-4bb8-a24e-1e035a961a3d" containerID="3e5c7455a8d5cdf18b6bc98ac567697bdd0889790042880233c18f5846d5212c" exitCode=255 Jul 07 17:23:05.674400 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.674382 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" event={"ID":"4d370278-be42-4bb8-a24e-1e035a961a3d","Type":"ContainerDied","Data":"3e5c7455a8d5cdf18b6bc98ac567697bdd0889790042880233c18f5846d5212c"} Jul 07 17:23:05.674442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.674433 2569 scope.go:117] "RemoveContainer" containerID="411f03b09715c01ccccd4f4c84238b4992510d5f3dc88d1a7552dc5961c1381b" Jul 07 17:23:05.674635 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.674615 2569 scope.go:117] "RemoveContainer" containerID="3e5c7455a8d5cdf18b6bc98ac567697bdd0889790042880233c18f5846d5212c" Jul 07 17:23:05.674876 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:05.674852 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2ffz_openshift-console-operator(4d370278-be42-4bb8-a24e-1e035a961a3d)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" podUID="4d370278-be42-4bb8-a24e-1e035a961a3d" Jul 07 17:23:05.675789 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.675690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" event={"ID":"157a2dab-3b23-4255-8c68-2dc658d24263","Type":"ContainerStarted","Data":"cfc91b2edd57c00c5ecd196a625aa8e135182ac78c4a5bc58a9833037e51dfea"} Jul 07 17:23:05.903748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:05.903719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:06.003060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:06.002979 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:06.018020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:06.017995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:06.018131 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:06.018073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:06.018131 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:06.018121 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:23:06.018204 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:06.018170 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:07.01815691 +0000 UTC m=+52.138361217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : configmap "kube-rbac-proxy" not found Jul 07 17:23:06.018262 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:06.018212 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:23:06.018318 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:06.018264 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:07.018251998 +0000 UTC m=+52.138456305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : secret "insights-runtime-extractor-tls" not found Jul 07 17:23:06.680956 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:06.680925 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:23:06.681407 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:06.681345 2569 scope.go:117] "RemoveContainer" containerID="3e5c7455a8d5cdf18b6bc98ac567697bdd0889790042880233c18f5846d5212c" Jul 07 17:23:06.681562 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:06.681535 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2ffz_openshift-console-operator(4d370278-be42-4bb8-a24e-1e035a961a3d)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" podUID="4d370278-be42-4bb8-a24e-1e035a961a3d" Jul 07 17:23:07.010663 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.010582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:07.027773 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.027741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:07.027922 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.027844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:07.027922 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:07.027859 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:23:07.027922 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:07.027915 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:09.027900365 +0000 UTC m=+54.148104673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : configmap "kube-rbac-proxy" not found Jul 07 17:23:07.028052 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:07.027986 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:23:07.028052 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:07.028047 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:09.028034915 +0000 UTC m=+54.148239227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : secret "insights-runtime-extractor-tls" not found Jul 07 17:23:07.535145 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.535118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qkd9s_595c820e-0463-4011-851d-f3dea5f189e6/dns-node-resolver/0.log" Jul 07 17:23:07.540782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.540757 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9tnx"] Jul 07 17:23:07.560213 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.560192 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9tnx"] Jul 07 17:23:07.560353 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.560276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.565872 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.565847 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 17:23:07.566007 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.565904 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 17:23:07.566007 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.565918 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 17:23:07.567414 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.567081 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 17:23:07.567414 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.567108 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-64h4z\"" Jul 07 17:23:07.586030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.586000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:07.633940 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.633912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f632176c-17fa-439a-ac08-7c79cf881b78-signing-cabundle\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.634074 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.634052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f632176c-17fa-439a-ac08-7c79cf881b78-signing-key\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.634146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.634128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnj8\" (UniqueName: \"kubernetes.io/projected/f632176c-17fa-439a-ac08-7c79cf881b78-kube-api-access-7bnj8\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.682863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.682833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:07.734899 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.734873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f632176c-17fa-439a-ac08-7c79cf881b78-signing-key\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.735032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.734936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnj8\" (UniqueName: \"kubernetes.io/projected/f632176c-17fa-439a-ac08-7c79cf881b78-kube-api-access-7bnj8\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.735032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.734996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f632176c-17fa-439a-ac08-7c79cf881b78-signing-cabundle\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.735725 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.735661 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f632176c-17fa-439a-ac08-7c79cf881b78-signing-cabundle\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.737335 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.737308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f632176c-17fa-439a-ac08-7c79cf881b78-signing-key\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.744197 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.744178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnj8\" (UniqueName: \"kubernetes.io/projected/f632176c-17fa-439a-ac08-7c79cf881b78-kube-api-access-7bnj8\") pod \"service-ca-758f599c5-r9tnx\" (UID: \"f632176c-17fa-439a-ac08-7c79cf881b78\") " pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:07.782984 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.782950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:07.869793 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:07.869774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-r9tnx" Jul 07 17:23:08.023147 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.023111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9tnx"] Jul 07 17:23:08.530024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.529998 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hnzsh_75eaed74-372b-4f2d-a80e-bfb1b0398601/node-ca/0.log" Jul 07 17:23:08.690355 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.690311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" event={"ID":"157a2dab-3b23-4255-8c68-2dc658d24263","Type":"ContainerStarted","Data":"c5dc10c31da872377af40384ba1e9ea8abb0b67179e5bcc6fd8dc539a41bf02f"} Jul 07 17:23:08.690788 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.690371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" event={"ID":"157a2dab-3b23-4255-8c68-2dc658d24263","Type":"ContainerStarted","Data":"dd1e4053600f0c35908b9060a941cfaa3524c24108b6d4d61aac424a304e6b9a"} Jul 07 17:23:08.691913 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.691888 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-r9tnx" event={"ID":"f632176c-17fa-439a-ac08-7c79cf881b78","Type":"ContainerStarted","Data":"5554d1ea8b8b6bfa5a5c29e2a959b4a767e0b5886c2515dfd20c21a2807d4c0e"} Jul 07 17:23:08.692062 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.691930 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-r9tnx" event={"ID":"f632176c-17fa-439a-ac08-7c79cf881b78","Type":"ContainerStarted","Data":"dc63d2162b5c7b1ba65ab9fbdc0f53e6161a613359e9d60763e19e525ba21e77"} Jul 07 17:23:08.716503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.716448 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jlfsf" podStartSLOduration=1.406086154 podStartE2EDuration="3.716428055s" totalImagesPullingTime="2.310341901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:08.715314372 +0000 UTC m=+53.835518704" watchObservedRunningTime="2026-07-07 17:23:08.716428055 +0000 UTC m=+53.836632385" Jul 07 17:23:08.721066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.721012 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:08.930449 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:08.930412 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqgft_5391d841-23bf-4e93-848e-b2788bc05d76/serve-healthcheck-canary/0.log" Jul 07 17:23:09.049206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.049172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:09.049385 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.049271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:09.049385 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:09.049361 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:23:09.049385 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:09.049363 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:23:09.049488 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:09.049411 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:13.049398785 +0000 UTC m=+58.169603093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : configmap "kube-rbac-proxy" not found Jul 07 17:23:09.049488 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:09.049423 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:13.04941741 +0000 UTC m=+58.169621719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : secret "insights-runtime-extractor-tls" not found Jul 07 17:23:09.130883 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.130842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-jlfsf_157a2dab-3b23-4255-8c68-2dc658d24263/migrator/0.log" Jul 07 17:23:09.330854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.330777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-jlfsf_157a2dab-3b23-4255-8c68-2dc658d24263/graceful-termination/0.log" Jul 07 17:23:09.532751 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.532711 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-jwwhp_5876c8dd-c233-475f-a608-85c8f137c6c7/kube-storage-version-migrator-operator/0.log" Jul 07 17:23:09.571666 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.571629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:09.704488 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.704452 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:09.775908 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.775873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:09.915100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:09.915067 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:10.017648 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:10.017562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:10.260038 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:10.260005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:10.858977 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:10.858943 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:11.069440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.069405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:11.069440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.069444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:23:11.069638 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.069548 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:23:11.069638 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.069570 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:23:11.069638 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.069594 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d65479c8-fcb4w: secret "image-registry-tls" not found Jul 07 17:23:11.069638 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.069610 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls podName:a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:27.069593955 +0000 UTC m=+72.189798263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mtbfz" (UID: "a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:23:11.069755 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.069644 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls podName:4d16c4a0-59f5-44ac-9051-d8edf4e0bb50 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:27.06962834 +0000 UTC m=+72.189832662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls") pod "image-registry-67d65479c8-fcb4w" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50") : secret "image-registry-tls" not found Jul 07 17:23:11.170971 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.170944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:11.171094 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.171060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:11.171094 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.171085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:23:11.171157 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.171130 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:27.171108837 +0000 UTC m=+72.291313160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : configmap references non-existent config key: service-ca.crt Jul 07 17:23:11.171221 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.171203 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:23:11.171221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.171210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:11.171301 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.171256 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls podName:c1ed0db7-bbde-4b36-9d77-85a4b6328a24 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:27.171240474 +0000 UTC m=+72.291444804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls") pod "dns-default-dxjt7" (UID: "c1ed0db7-bbde-4b36-9d77-85a4b6328a24") : secret "dns-default-metrics-tls" not found Jul 07 17:23:11.171365 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.171351 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:23:11.171409 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:11.171399 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs podName:bb2371ec-c2d8-4094-a9e4-939d0601b6d5 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:27.17138731 +0000 UTC m=+72.291591631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs") pod "router-default-784748979c-znp9d" (UID: "bb2371ec-c2d8-4094-a9e4-939d0601b6d5") : secret "router-metrics-certs-default" not found Jul 07 17:23:11.173587 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.173568 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2807ef90-5a4a-4bc3-ac29-240205e2890b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ld856\" (UID: \"2807ef90-5a4a-4bc3-ac29-240205e2890b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:23:11.463975 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.463897 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" Jul 07 17:23:11.590357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.590308 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-r9tnx" podStartSLOduration=4.590264595 podStartE2EDuration="4.590264595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:08.738690757 +0000 UTC m=+53.858895088" watchObservedRunningTime="2026-07-07 17:23:11.590264595 +0000 UTC m=+56.710468924" Jul 07 17:23:11.591092 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.591071 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856"] Jul 07 17:23:11.642023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.641993 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:11.708768 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.708734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" event={"ID":"2807ef90-5a4a-4bc3-ac29-240205e2890b","Type":"ContainerStarted","Data":"71bee518f5a241ee6fa840eb314ef691f6fede29df863976812c3ec1a6823d5c"} Jul 07 17:23:11.800974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.800890 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:11.902075 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:11.902034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:12.070680 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:12.070597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:12.354692 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:12.354606 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:12.459864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:12.459827 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:12.561793 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:12.561747 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:13.088760 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.088722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:13.088760 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.088765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.088831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088877 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088922 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088928 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088954 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert podName:4926c65f-8c54-4aa5-9ea1-b9e59b52b44b nodeName:}" failed. No retries permitted until 2026-07-07 17:23:29.088935699 +0000 UTC m=+74.209140028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ch6fp" (UID: "4926c65f-8c54-4aa5-9ea1-b9e59b52b44b") : secret "networking-console-plugin-cert" not found Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088969 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:21.08896313 +0000 UTC m=+66.209167437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : secret "insights-runtime-extractor-tls" not found Jul 07 17:23:13.089211 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:23:13.088981 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm podName:5e3a6b56-8f0f-4368-9afc-d3d485c54416 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:21.088975022 +0000 UTC m=+66.209179331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jl4g" (UID: "5e3a6b56-8f0f-4368-9afc-d3d485c54416") : configmap "kube-rbac-proxy" not found Jul 07 17:23:13.243653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.243614 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:13.345797 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.345722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:13.479429 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.479391 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:13.647816 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:13.647780 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:14.054360 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.054192 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:14.140856 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.140818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:14.645507 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.645475 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:14.722782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.722746 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" event={"ID":"2807ef90-5a4a-4bc3-ac29-240205e2890b","Type":"ContainerStarted","Data":"9b1cfe60faee138428bfb54324e1409e2ef6c9f42a5c0e4cceecfe6d287a359e"} Jul 07 17:23:14.722947 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.722798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" event={"ID":"2807ef90-5a4a-4bc3-ac29-240205e2890b","Type":"ContainerStarted","Data":"582bb03e66efacb798412eafa86dd98f54d198caf1cab2927c3b5f2de952ebfa"} Jul 07 17:23:14.743633 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:14.743583 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ld856" podStartSLOduration=25.092874127 podStartE2EDuration="27.743566037s" totalImagesPullingTime="2.65069191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:14.742681519 +0000 UTC m=+59.862885850" watchObservedRunningTime="2026-07-07 17:23:14.743566037 +0000 UTC m=+59.863770367" Jul 07 17:23:15.238211 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.238175 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:15.339646 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.339597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:15.440605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.440568 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:15.519722 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.519638 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:15.587024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.586990 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:15.587188 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.587044 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:15.587416 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.587402 2569 scope.go:117] "RemoveContainer" containerID="3e5c7455a8d5cdf18b6bc98ac567697bdd0889790042880233c18f5846d5212c" Jul 07 17:23:15.668364 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.668328 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:15.729317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.728573 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:23:15.729317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.728700 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" event={"ID":"4d370278-be42-4bb8-a24e-1e035a961a3d","Type":"ContainerStarted","Data":"dbe5724d74c1efd14120e8af4fcb3393fbd34c0d465b8f61c338c7b4c6675690"} Jul 07 17:23:15.729317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.729145 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:15.730475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.730448 2569 patch_prober.go:28] interesting pod/console-operator-68b757fb67-p2ffz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Jul 07 17:23:15.730602 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.730492 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" podUID="4d370278-be42-4bb8-a24e-1e035a961a3d" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Jul 07 17:23:15.752047 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.752004 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" podStartSLOduration=21.040979265 podStartE2EDuration="28.751990279s" totalImagesPullingTime="7.711011014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:15.750666474 +0000 UTC m=+60.870870804" watchObservedRunningTime="2026-07-07 17:23:15.751990279 +0000 UTC m=+60.872194608" Jul 07 17:23:15.757970 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:15.757948 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.078017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.077978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.314525 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.314489 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.415827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.415768 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.515709 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.515674 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.649537 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.649492 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:16.736063 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:16.735994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-p2ffz" Jul 07 17:23:17.011651 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.011573 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.193084 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.193030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.334740 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.334664 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.433822 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.433785 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.513215 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.513160 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.645728 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.645696 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:17.814013 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:17.813976 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.033062 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.032949 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.113228 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.113189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.182516 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.182484 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.289660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.289565 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.825039 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.825007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:18.925846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:18.925799 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:19.444454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:19.444420 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:19.653755 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:19.653725 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:19.852596 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:19.852528 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:20.064029 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.063995 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:20.289745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.289713 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:20.490069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.490040 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:20.600371 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.600297 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9nxj" Jul 07 17:23:20.647971 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.647935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:20.806863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:20.806838 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:21.162966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.162929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:21.163370 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.163015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:21.163370 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.163040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:23:21.163561 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.163531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e3a6b56-8f0f-4368-9afc-d3d485c54416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:21.165416 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.165400 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e3a6b56-8f0f-4368-9afc-d3d485c54416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jl4g\" (UID: \"5e3a6b56-8f0f-4368-9afc-d3d485c54416\") " pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:21.166128 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.166116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:23:21.175467 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.175447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87d5a160-779e-4370-b6fb-96a892b2dbca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x4x\" (UID: \"87d5a160-779e-4370-b6fb-96a892b2dbca\") " pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:23:21.262879 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.262850 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2gj6\"" Jul 07 17:23:21.264120 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.264101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:23:21.266037 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.266018 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x4x" Jul 07 17:23:21.266728 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.266712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:23:21.269878 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.269856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jl4g" Jul 07 17:23:21.276825 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.276801 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3aed3c-1f0d-466a-834e-86aa02f6e884-metrics-certs\") pod \"network-metrics-daemon-9sfzm\" (UID: \"ca3aed3c-1f0d-466a-834e-86aa02f6e884\") " pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:23:21.364804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.364776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:23:21.367489 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.367462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp8q\" (UniqueName: \"kubernetes.io/projected/20090afb-e3c1-429c-8a1c-bed594800579-kube-api-access-2gp8q\") pod \"network-check-target-88r7j\" (UID: \"20090afb-e3c1-429c-8a1c-bed594800579\") " pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:23:21.413113 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.413067 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jl4g"] Jul 07 17:23:21.414939 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:21.414913 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3a6b56_8f0f_4368_9afc_d3d485c54416.slice/crio-4e1d95463b6d48b71314d4f503238f23356f6cca647cd39a4dee967ffe1bf395 WatchSource:0}: Error finding container 4e1d95463b6d48b71314d4f503238f23356f6cca647cd39a4dee967ffe1bf395: Status 404 returned error can't find the container with id 4e1d95463b6d48b71314d4f503238f23356f6cca647cd39a4dee967ffe1bf395 Jul 07 17:23:21.433905 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.433883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8x4x"] Jul 07 17:23:21.436692 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:21.436664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d5a160_779e_4370_b6fb_96a892b2dbca.slice/crio-28a72deb116f0afd1651cdaf1a6a0932081904515ffb4d7ae8cd58a04e94f234 WatchSource:0}: Error finding container 28a72deb116f0afd1651cdaf1a6a0932081904515ffb4d7ae8cd58a04e94f234: Status 404 returned error can't find the container with id 28a72deb116f0afd1651cdaf1a6a0932081904515ffb4d7ae8cd58a04e94f234 Jul 07 17:23:21.558689 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.558657 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7rcfs\"" Jul 07 17:23:21.563173 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.563156 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmhtn\"" Jul 07 17:23:21.566319 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.566308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sfzm" Jul 07 17:23:21.571707 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.571688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:23:21.709110 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.709089 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9sfzm"] Jul 07 17:23:21.711137 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:21.711112 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3aed3c_1f0d_466a_834e_86aa02f6e884.slice/crio-037ca0d01446160ee5921766d56b71e156aad5945cabd38549ccb4b19b4dac9b WatchSource:0}: Error finding container 037ca0d01446160ee5921766d56b71e156aad5945cabd38549ccb4b19b4dac9b: Status 404 returned error can't find the container with id 037ca0d01446160ee5921766d56b71e156aad5945cabd38549ccb4b19b4dac9b Jul 07 17:23:21.724376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.724353 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88r7j"] Jul 07 17:23:21.727083 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:21.727055 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20090afb_e3c1_429c_8a1c_bed594800579.slice/crio-e55e45d9d7156c48f84a8685b3bee8abcec86d462c94b1a03d2459b47236c135 WatchSource:0}: Error finding container e55e45d9d7156c48f84a8685b3bee8abcec86d462c94b1a03d2459b47236c135: Status 404 returned error can't find the container with id e55e45d9d7156c48f84a8685b3bee8abcec86d462c94b1a03d2459b47236c135 Jul 07 17:23:21.749150 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.749122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sfzm" event={"ID":"ca3aed3c-1f0d-466a-834e-86aa02f6e884","Type":"ContainerStarted","Data":"037ca0d01446160ee5921766d56b71e156aad5945cabd38549ccb4b19b4dac9b"} Jul 07 17:23:21.750165 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.750143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8x4x" event={"ID":"87d5a160-779e-4370-b6fb-96a892b2dbca","Type":"ContainerStarted","Data":"28a72deb116f0afd1651cdaf1a6a0932081904515ffb4d7ae8cd58a04e94f234"} Jul 07 17:23:21.751184 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.751163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88r7j" event={"ID":"20090afb-e3c1-429c-8a1c-bed594800579","Type":"ContainerStarted","Data":"e55e45d9d7156c48f84a8685b3bee8abcec86d462c94b1a03d2459b47236c135"} Jul 07 17:23:21.752163 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:21.752145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jl4g" event={"ID":"5e3a6b56-8f0f-4368-9afc-d3d485c54416","Type":"ContainerStarted","Data":"4e1d95463b6d48b71314d4f503238f23356f6cca647cd39a4dee967ffe1bf395"} Jul 07 17:23:22.769472 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:22.769433 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88r7j" event={"ID":"20090afb-e3c1-429c-8a1c-bed594800579","Type":"ContainerStarted","Data":"34e8c7d5e21c381e4d8b1b4dcf65de70f5188350e68f38147ec08f1f6b701858"} Jul 07 17:23:22.769927 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:22.769702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:23:22.791200 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:22.790546 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-88r7j" podStartSLOduration=67.790529405 podStartE2EDuration="1m7.790529405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:22.788726611 +0000 UTC m=+67.908930953" watchObservedRunningTime="2026-07-07 17:23:22.790529405 +0000 UTC m=+67.910733924" Jul 07 17:23:24.450605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:24.450562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:24.536209 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:24.536180 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:26.786882 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:26.786848 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jl4g" event={"ID":"5e3a6b56-8f0f-4368-9afc-d3d485c54416","Type":"ContainerStarted","Data":"51bb86b6e89d7e6791212d52b4977bd4912e59f3fbe376047680f0591dcf7134"} Jul 07 17:23:26.787269 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:26.786912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jl4g" event={"ID":"5e3a6b56-8f0f-4368-9afc-d3d485c54416","Type":"ContainerStarted","Data":"5217cd227ad6282b161ae9610027c5313b5f5de6e2c7f0878eed868ab28a639a"} Jul 07 17:23:26.788688 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:26.788663 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sfzm" event={"ID":"ca3aed3c-1f0d-466a-834e-86aa02f6e884","Type":"ContainerStarted","Data":"e026c25723cdff6abf23ce2cd3421a7cf070e01b632f3ea192b2228d89ba9bbd"} Jul 07 17:23:26.788800 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:26.788708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sfzm" event={"ID":"ca3aed3c-1f0d-466a-834e-86aa02f6e884","Type":"ContainerStarted","Data":"c8bf7cda180f901b9d3e14a605e67e76f92b95ab1baa62c2bc27af1526038dcc"} Jul 07 17:23:26.830610 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:26.830532 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9sfzm" podStartSLOduration=67.736815269 podStartE2EDuration="1m11.830516507s" totalImagesPullingTime="4.093701238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:26.829787086 +0000 UTC m=+71.949991418" watchObservedRunningTime="2026-07-07 17:23:26.830516507 +0000 UTC m=+71.950720837" Jul 07 17:23:27.120017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.119934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:27.120017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.119986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:23:27.122934 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.122905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mtbfz\" (UID: \"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:23:27.123146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.123125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"image-registry-67d65479c8-fcb4w\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:27.220969 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.220934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:27.221136 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.220991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:27.221136 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.221046 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:27.221764 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.221734 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-service-ca-bundle\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:27.223615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.223590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1ed0db7-bbde-4b36-9d77-85a4b6328a24-metrics-tls\") pod \"dns-default-dxjt7\" (UID: \"c1ed0db7-bbde-4b36-9d77-85a4b6328a24\") " pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:27.223846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.223830 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2371ec-c2d8-4094-a9e4-939d0601b6d5-metrics-certs\") pod \"router-default-784748979c-znp9d\" (UID: \"bb2371ec-c2d8-4094-a9e4-939d0601b6d5\") " pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:27.302361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.302326 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5jqz\"" Jul 07 17:23:27.309954 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.309926 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:27.325017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.324831 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7lgkq\"" Jul 07 17:23:27.331660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.331635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" Jul 07 17:23:27.355708 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.355605 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w6x9j\"" Jul 07 17:23:27.362882 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.362718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:27.377472 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.377449 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pgv6p\"" Jul 07 17:23:27.384973 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.384649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:27.499037 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.498944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:23:27.538276 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.537725 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz"] Jul 07 17:23:27.569661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.569637 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-784748979c-znp9d"] Jul 07 17:23:27.595305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:27.595203 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxjt7"] Jul 07 17:23:28.993638 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:28.993598 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d16c4a0_59f5_44ac_9051_d8edf4e0bb50.slice/crio-a5225d080632de019bbd6abde9c2104503389744259ceda56ebb34b014ba42ec WatchSource:0}: Error finding container a5225d080632de019bbd6abde9c2104503389744259ceda56ebb34b014ba42ec: Status 404 returned error can't find the container with id a5225d080632de019bbd6abde9c2104503389744259ceda56ebb34b014ba42ec Jul 07 17:23:28.994753 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:28.994591 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ed0db7_bbde_4b36_9d77_85a4b6328a24.slice/crio-6dce1201a09f27fde202f85ff552a21c93f0e91d2fb5f2bbf654e866c5dae63e WatchSource:0}: Error finding container 6dce1201a09f27fde202f85ff552a21c93f0e91d2fb5f2bbf654e866c5dae63e: Status 404 returned error can't find the container with id 6dce1201a09f27fde202f85ff552a21c93f0e91d2fb5f2bbf654e866c5dae63e Jul 07 17:23:28.995160 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:28.995138 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2371ec_c2d8_4094_a9e4_939d0601b6d5.slice/crio-e61fd3bb88cbda8bda8654dd2fd98ccf1cc6b30faa33b4d482bd08e52980f7d4 WatchSource:0}: Error finding container e61fd3bb88cbda8bda8654dd2fd98ccf1cc6b30faa33b4d482bd08e52980f7d4: Status 404 returned error can't find the container with id e61fd3bb88cbda8bda8654dd2fd98ccf1cc6b30faa33b4d482bd08e52980f7d4 Jul 07 17:23:28.996059 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:28.996040 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cfcfec_4f6b_41e9_8ccb_4d9e93ebd6b4.slice/crio-a2bfee731b9d47c29f4e8c6b84c3109df417a933df0e24142f0652656797bd1d WatchSource:0}: Error finding container a2bfee731b9d47c29f4e8c6b84c3109df417a933df0e24142f0652656797bd1d: Status 404 returned error can't find the container with id a2bfee731b9d47c29f4e8c6b84c3109df417a933df0e24142f0652656797bd1d Jul 07 17:23:29.141673 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.141638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:29.144594 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.144563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4926c65f-8c54-4aa5-9ea1-b9e59b52b44b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ch6fp\" (UID: \"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:29.263202 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.263123 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9xdwm\"" Jul 07 17:23:29.270737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.270696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" Jul 07 17:23:29.801878 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.801837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" event={"ID":"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4","Type":"ContainerStarted","Data":"a2bfee731b9d47c29f4e8c6b84c3109df417a933df0e24142f0652656797bd1d"} Jul 07 17:23:29.803161 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.803097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxjt7" event={"ID":"c1ed0db7-bbde-4b36-9d77-85a4b6328a24","Type":"ContainerStarted","Data":"6dce1201a09f27fde202f85ff552a21c93f0e91d2fb5f2bbf654e866c5dae63e"} Jul 07 17:23:29.804494 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.804465 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-784748979c-znp9d" event={"ID":"bb2371ec-c2d8-4094-a9e4-939d0601b6d5","Type":"ContainerStarted","Data":"e61fd3bb88cbda8bda8654dd2fd98ccf1cc6b30faa33b4d482bd08e52980f7d4"} Jul 07 17:23:29.805709 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:29.805685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" event={"ID":"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50","Type":"ContainerStarted","Data":"a5225d080632de019bbd6abde9c2104503389744259ceda56ebb34b014ba42ec"} Jul 07 17:23:30.646565 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.645396 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp"] Jul 07 17:23:30.812212 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.811869 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8x4x" event={"ID":"87d5a160-779e-4370-b6fb-96a892b2dbca","Type":"ContainerStarted","Data":"8c14382222f9726dc2d6b5e38c51a951ec089b3a973a02e390a2d4e94c5d9fcd"} Jul 07 17:23:30.813853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.813772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" event={"ID":"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b","Type":"ContainerStarted","Data":"246c7fe1ac0567b8b38bf2331dcc7137ca9805a91af6715eef453dccf1a0be96"} Jul 07 17:23:30.816247 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.816222 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-784748979c-znp9d" event={"ID":"bb2371ec-c2d8-4094-a9e4-939d0601b6d5","Type":"ContainerStarted","Data":"27d46187b0bd001cf23de894c2eb35244433cc39d1df4935e6f3afdc982660ff"} Jul 07 17:23:30.820418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.819752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" event={"ID":"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50","Type":"ContainerStarted","Data":"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a"} Jul 07 17:23:30.820418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.820277 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:30.823802 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.823276 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jl4g" event={"ID":"5e3a6b56-8f0f-4368-9afc-d3d485c54416","Type":"ContainerStarted","Data":"d34ba0d2d52f34db362bd45ae95b4ff262a95a28fe5e3f0630f369832c616be9"} Jul 07 17:23:30.834232 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.834184 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h8x4x" podStartSLOduration=66.782884367 podStartE2EDuration="1m15.834165673s" totalImagesPullingTime="9.051281306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:30.832347094 +0000 UTC m=+75.952551426" watchObservedRunningTime="2026-07-07 17:23:30.834165673 +0000 UTC m=+75.954370004" Jul 07 17:23:30.903796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.903728 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" podStartSLOduration=75.903705946 podStartE2EDuration="1m15.903705946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:30.87498331 +0000 UTC m=+75.995187640" watchObservedRunningTime="2026-07-07 17:23:30.903705946 +0000 UTC m=+76.023910277" Jul 07 17:23:30.904490 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.904447 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5jl4g" podStartSLOduration=16.970177485 podStartE2EDuration="25.904432892s" totalImagesPullingTime="8.934255407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:23:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:30.903545864 +0000 UTC m=+76.023750194" watchObservedRunningTime="2026-07-07 17:23:30.904432892 +0000 UTC m=+76.024637226" Jul 07 17:23:30.930475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:30.930426 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-784748979c-znp9d" podStartSLOduration=43.930410999 podStartE2EDuration="43.930410999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:30.929640037 +0000 UTC m=+76.049844370" watchObservedRunningTime="2026-07-07 17:23:30.930410999 +0000 UTC m=+76.050615329" Jul 07 17:23:31.037472 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.037436 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:31.137937 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.137880 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:31.363551 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.363504 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:31.366577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.366550 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:31.731787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.731438 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6"] Jul 07 17:23:31.746027 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.745963 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4"] Jul 07 17:23:31.746185 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.746106 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.751865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.751458 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 17:23:31.751865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.751687 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 17:23:31.752244 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.752222 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 17:23:31.752539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.752519 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 17:23:31.752746 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.752732 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 17:23:31.756201 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.752817 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 17:23:31.756201 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.753057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 17:23:31.758931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.758791 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx"] Jul 07 17:23:31.759718 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.759066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.764817 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.764795 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 17:23:31.771123 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.770574 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx"] Jul 07 17:23:31.771123 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.770619 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4"] Jul 07 17:23:31.771123 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.770725 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:31.772771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.772702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6"] Jul 07 17:23:31.773499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.773479 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 17:23:31.773756 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.773737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lgvkv\"" Jul 07 17:23:31.830097 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.830066 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:31.830994 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.830944 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-784748979c-znp9d" Jul 07 17:23:31.836566 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.836545 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-zpbpz"] Jul 07 17:23:31.846240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.846214 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:31.848859 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.848836 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:31.849027 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.849009 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b4htg\"" Jul 07 17:23:31.849105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.849020 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 17:23:31.849150 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.849138 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 17:23:31.857505 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.857481 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zpbpz"] Jul 07 17:23:31.866747 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.866893 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccb0125-72bd-4658-94bf-d990bef53d80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:31.866893 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-klusterlet-config\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.866893 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.867023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jz5w\" (UniqueName: \"kubernetes.io/projected/2ccb0125-72bd-4658-94bf-d990bef53d80-kube-api-access-8jz5w\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:31.867023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.867023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.866957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.867023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.867015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-tmp\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.867181 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.867044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnx4\" (UniqueName: \"kubernetes.io/projected/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-kube-api-access-tnnx4\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.867181 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.867064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc35cc2-af98-4b48-b032-314738bb4895-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.867181 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.867085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55cq\" (UniqueName: \"kubernetes.io/projected/dcc35cc2-af98-4b48-b032-314738bb4895-kube-api-access-q55cq\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968222 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-tmp\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.968431 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnx4\" (UniqueName: \"kubernetes.io/projected/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-kube-api-access-tnnx4\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.968431 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc35cc2-af98-4b48-b032-314738bb4895-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968431 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55cq\" (UniqueName: \"kubernetes.io/projected/dcc35cc2-af98-4b48-b032-314738bb4895-kube-api-access-q55cq\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968602 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968602 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccb0125-72bd-4658-94bf-d990bef53d80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:31.968721 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968609 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-tmp\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.968721 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-klusterlet-config\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.968721 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8knb\" (UniqueName: \"kubernetes.io/projected/8912dfbf-eba0-43ed-895a-06b623bacc7e-kube-api-access-z8knb\") pod \"downloads-685bfb5f64-zpbpz\" (UID: \"8912dfbf-eba0-43ed-895a-06b623bacc7e\") " pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:31.968854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jz5w\" (UniqueName: \"kubernetes.io/projected/2ccb0125-72bd-4658-94bf-d990bef53d80-kube-api-access-8jz5w\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:31.968854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.968995 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.968876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.969331 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.969276 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc35cc2-af98-4b48-b032-314738bb4895-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.973126 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.973100 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.973408 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.973380 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.973526 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.973477 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-ca\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.974097 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.974074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-klusterlet-config\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:31.974275 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.974256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcc35cc2-af98-4b48-b032-314738bb4895-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:31.974789 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:31.974759 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccb0125-72bd-4658-94bf-d990bef53d80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:32.004060 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.003953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jz5w\" (UniqueName: \"kubernetes.io/projected/2ccb0125-72bd-4658-94bf-d990bef53d80-kube-api-access-8jz5w\") pod \"managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx\" (UID: \"2ccb0125-72bd-4658-94bf-d990bef53d80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:32.004194 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.004176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55cq\" (UniqueName: \"kubernetes.io/projected/dcc35cc2-af98-4b48-b032-314738bb4895-kube-api-access-q55cq\") pod \"cluster-proxy-proxy-agent-57bcc96f56-6kgf6\" (UID: \"dcc35cc2-af98-4b48-b032-314738bb4895\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:32.006104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.006029 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.006104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.006069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnx4\" (UniqueName: \"kubernetes.io/projected/21ac3b4d-ea6f-4bcc-9288-39e2d0190716-kube-api-access-tnnx4\") pod \"klusterlet-addon-workmgr-844c74b46c-w4gx4\" (UID: \"21ac3b4d-ea6f-4bcc-9288-39e2d0190716\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:32.069738 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.069705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8knb\" (UniqueName: \"kubernetes.io/projected/8912dfbf-eba0-43ed-895a-06b623bacc7e-kube-api-access-z8knb\") pod \"downloads-685bfb5f64-zpbpz\" (UID: \"8912dfbf-eba0-43ed-895a-06b623bacc7e\") " pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:32.080674 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.080646 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" Jul 07 17:23:32.090068 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.090045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8knb\" (UniqueName: \"kubernetes.io/projected/8912dfbf-eba0-43ed-895a-06b623bacc7e-kube-api-access-z8knb\") pod \"downloads-685bfb5f64-zpbpz\" (UID: \"8912dfbf-eba0-43ed-895a-06b623bacc7e\") " pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:32.092389 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.092356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:32.100258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.100239 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" Jul 07 17:23:32.167206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.167179 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:32.190074 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.190040 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.391342 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.391303 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.490753 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.490715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.555100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.555059 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.637903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.637867 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:32.701962 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:32.701885 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:33.842036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.841986 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4"] Jul 07 17:23:33.846382 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.845569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" event={"ID":"4926c65f-8c54-4aa5-9ea1-b9e59b52b44b","Type":"ContainerStarted","Data":"b5c4ef79061f9b34b2bb6051a893529dc6e5d27bb7643e1f133ab82692031c41"} Jul 07 17:23:33.847693 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:33.847542 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ac3b4d_ea6f_4bcc_9288_39e2d0190716.slice/crio-b687b354bd04ff52febd366dc14df1c6f56bdb7f5a7a3d3bc0c98f66eca5ea31 WatchSource:0}: Error finding container b687b354bd04ff52febd366dc14df1c6f56bdb7f5a7a3d3bc0c98f66eca5ea31: Status 404 returned error can't find the container with id b687b354bd04ff52febd366dc14df1c6f56bdb7f5a7a3d3bc0c98f66eca5ea31 Jul 07 17:23:33.848610 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.848381 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" event={"ID":"a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4","Type":"ContainerStarted","Data":"55ab684310f4df0c5c85fc6a23738b1f60d75d488a17ed393f664360b378b171"} Jul 07 17:23:33.871364 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.871273 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6"] Jul 07 17:23:33.875620 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:33.875589 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc35cc2_af98_4b48_b032_314738bb4895.slice/crio-e18d5e36c440e89b65de5a8cc71982a2d133b9d2b577da8374825b2810e82810 WatchSource:0}: Error finding container e18d5e36c440e89b65de5a8cc71982a2d133b9d2b577da8374825b2810e82810: Status 404 returned error can't find the container with id e18d5e36c440e89b65de5a8cc71982a2d133b9d2b577da8374825b2810e82810 Jul 07 17:23:33.878167 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.878041 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mtbfz" podStartSLOduration=42.23156916 podStartE2EDuration="46.878025928s" totalImagesPullingTime="4.646456768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:33.877222639 +0000 UTC m=+78.997426970" watchObservedRunningTime="2026-07-07 17:23:33.878025928 +0000 UTC m=+78.998230267" Jul 07 17:23:33.892926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:33.892898 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx"] Jul 07 17:23:33.896277 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:33.896253 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ccb0125_72bd_4658_94bf_d990bef53d80.slice/crio-4719b7e83adaf788c4fbb687895540913d11c757e61fe7f2582eecf88bbf035e WatchSource:0}: Error finding container 4719b7e83adaf788c4fbb687895540913d11c757e61fe7f2582eecf88bbf035e: Status 404 returned error can't find the container with id 4719b7e83adaf788c4fbb687895540913d11c757e61fe7f2582eecf88bbf035e Jul 07 17:23:34.095161 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.095134 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zpbpz"] Jul 07 17:23:34.097331 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:34.097271 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8912dfbf_eba0_43ed_895a_06b623bacc7e.slice/crio-e34ac79fc7f299578f9a56d2c1ca0b2b7801c63fa15743285f284b5d8e091645 WatchSource:0}: Error finding container e34ac79fc7f299578f9a56d2c1ca0b2b7801c63fa15743285f284b5d8e091645: Status 404 returned error can't find the container with id e34ac79fc7f299578f9a56d2c1ca0b2b7801c63fa15743285f284b5d8e091645 Jul 07 17:23:34.130866 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.130827 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.234853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.234772 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.325523 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.325452 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.330414 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.330387 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t"] Jul 07 17:23:34.341623 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.341596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:34.345182 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.344685 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4kkg\"" Jul 07 17:23:34.345182 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.345082 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 17:23:34.346710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.346653 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t"] Jul 07 17:23:34.455319 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.455222 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.492122 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.492038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01e94d75-ad29-4c3d-af53-95489acfc511-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-f9x6t\" (UID: \"01e94d75-ad29-4c3d-af53-95489acfc511\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:34.553132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.553090 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.593114 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.593078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01e94d75-ad29-4c3d-af53-95489acfc511-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-f9x6t\" (UID: \"01e94d75-ad29-4c3d-af53-95489acfc511\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:34.597238 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.597212 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01e94d75-ad29-4c3d-af53-95489acfc511-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-f9x6t\" (UID: \"01e94d75-ad29-4c3d-af53-95489acfc511\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:34.654619 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.654579 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.657231 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.657207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:34.731067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.730966 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:34.833026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.832996 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t"] Jul 07 17:23:34.856022 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.855989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxjt7" event={"ID":"c1ed0db7-bbde-4b36-9d77-85a4b6328a24","Type":"ContainerStarted","Data":"fb54834bcc1bd57239b910109ed8fd56441232af61b62612d46cfdc17184890c"} Jul 07 17:23:34.856453 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.856052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxjt7" event={"ID":"c1ed0db7-bbde-4b36-9d77-85a4b6328a24","Type":"ContainerStarted","Data":"a2350aeecf3225a9df9edcea0d1e9b0a1b0581c790eabc59710a1219cba4dcc7"} Jul 07 17:23:34.856453 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.856103 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:34.857405 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.857380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" event={"ID":"01e94d75-ad29-4c3d-af53-95489acfc511","Type":"ContainerStarted","Data":"83a0a53a1eae8c80809932a83b4ee0372ef8ffcda82aebb319a03e6c60281b11"} Jul 07 17:23:34.858666 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.858637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" event={"ID":"21ac3b4d-ea6f-4bcc-9288-39e2d0190716","Type":"ContainerStarted","Data":"b687b354bd04ff52febd366dc14df1c6f56bdb7f5a7a3d3bc0c98f66eca5ea31"} Jul 07 17:23:34.859871 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.859830 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" event={"ID":"2ccb0125-72bd-4658-94bf-d990bef53d80","Type":"ContainerStarted","Data":"4719b7e83adaf788c4fbb687895540913d11c757e61fe7f2582eecf88bbf035e"} Jul 07 17:23:34.861009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.860988 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" event={"ID":"dcc35cc2-af98-4b48-b032-314738bb4895","Type":"ContainerStarted","Data":"e18d5e36c440e89b65de5a8cc71982a2d133b9d2b577da8374825b2810e82810"} Jul 07 17:23:34.862641 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.862619 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zpbpz" event={"ID":"8912dfbf-eba0-43ed-895a-06b623bacc7e","Type":"ContainerStarted","Data":"e34ac79fc7f299578f9a56d2c1ca0b2b7801c63fa15743285f284b5d8e091645"} Jul 07 17:23:34.878031 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:34.877976 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dxjt7" podStartSLOduration=35.225120427 podStartE2EDuration="39.877957393s" totalImagesPullingTime="4.652836966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:34.877365406 +0000 UTC m=+79.997569737" watchObservedRunningTime="2026-07-07 17:23:34.877957393 +0000 UTC m=+79.998161725" Jul 07 17:23:35.386121 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:35.385792 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:35.747732 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:35.747623 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:35.852385 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:35.852303 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:35.944376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:35.944314 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:36.039169 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.038728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:36.119164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.119128 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:36.445034 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.444958 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ch6fp" podStartSLOduration=36.453771294 podStartE2EDuration="39.444936212s" totalImagesPullingTime="2.991164918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:34.898860767 +0000 UTC m=+80.019065097" watchObservedRunningTime="2026-07-07 17:23:36.444936212 +0000 UTC m=+81.565140543" Jul 07 17:23:36.445966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.445911 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:23:36.482321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.482272 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:23:36.482440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.482350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.487033 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487009 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zm2j4\"" Jul 07 17:23:36.487172 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487069 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 17:23:36.487239 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487223 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 17:23:36.487354 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487337 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 17:23:36.487577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487561 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 17:23:36.487772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.487756 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 17:23:36.514070 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.513918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.514070 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.513957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.514070 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.514023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.514347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.514088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.514347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.514127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfpg\" (UniqueName: \"kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.514347 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.514158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.555632 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.555582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:36.615314 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.614802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfpg\" (UniqueName: \"kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.615314 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.614853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.615314 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.614895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.615314 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.614919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.615960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.615935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.616063 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.615946 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.616126 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.616065 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.616126 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.616109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.617269 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.617160 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.622514 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.622489 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.635595 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.635569 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfpg\" (UniqueName: \"kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.636035 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.636009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config\") pod \"console-f85887fb8-kbnnk\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.637432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.637405 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:36.797447 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.797374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:36.952675 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:36.952639 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.053793 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.053700 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.142463 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.142428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.309503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.309403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.403644 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.403609 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.655912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.655832 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.739947 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.739890 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.842277 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.842242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:37.941801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:37.941704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:38.153411 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:38.153330 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:38.252443 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:38.252363 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:39.221659 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:39.221613 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:39.303725 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:39.303685 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:39.484271 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:39.484183 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:39.580133 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:39.580097 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:40.049868 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:40.049728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:40.123039 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:40.123001 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:41.032171 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.032111 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:41.136071 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.135789 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:41.246358 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.246249 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:23:41.251448 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:41.251407 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbc0b73_7f3c_460c_91d5_3b0140aef6c5.slice/crio-0550d48f28cf99ba6f97bfdd462772d920c417375b70174a6af7e0ed0d58de2e WatchSource:0}: Error finding container 0550d48f28cf99ba6f97bfdd462772d920c417375b70174a6af7e0ed0d58de2e: Status 404 returned error can't find the container with id 0550d48f28cf99ba6f97bfdd462772d920c417375b70174a6af7e0ed0d58de2e Jul 07 17:23:41.276452 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.276426 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:41.380701 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.380633 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:41.901173 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.901137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f85887fb8-kbnnk" event={"ID":"acbc0b73-7f3c-460c-91d5-3b0140aef6c5","Type":"ContainerStarted","Data":"0550d48f28cf99ba6f97bfdd462772d920c417375b70174a6af7e0ed0d58de2e"} Jul 07 17:23:41.902652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.902626 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" event={"ID":"01e94d75-ad29-4c3d-af53-95489acfc511","Type":"ContainerStarted","Data":"fea09550d67192c5660eef0d223321c40b713106b65cb9ac43d6372e99416b7b"} Jul 07 17:23:41.902796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.902768 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:41.904158 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.904133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" event={"ID":"21ac3b4d-ea6f-4bcc-9288-39e2d0190716","Type":"ContainerStarted","Data":"b0a18c33f591d4fb6069bad6912bb7a068e03ea27ed1a104c2376d5c24e0f72c"} Jul 07 17:23:41.904311 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.904297 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:41.905815 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.905796 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" Jul 07 17:23:41.905909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.905889 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" event={"ID":"2ccb0125-72bd-4658-94bf-d990bef53d80","Type":"ContainerStarted","Data":"609c7df725ca0ef32495a8a851ac0f96cd7093686beb355657000508a41ab6a4"} Jul 07 17:23:41.907925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.907903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" event={"ID":"dcc35cc2-af98-4b48-b032-314738bb4895","Type":"ContainerStarted","Data":"7959eb76f2ae5c096fffdd015c0c0e27d6c19ddb144e79abb46e93e9d803c6b0"} Jul 07 17:23:41.908008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.907933 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" event={"ID":"dcc35cc2-af98-4b48-b032-314738bb4895","Type":"ContainerStarted","Data":"4f3fc81bc3a5419b7585a8b29dc49693f26a135711cbc4c2c42a465ee16bce88"} Jul 07 17:23:41.908008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.907946 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" event={"ID":"dcc35cc2-af98-4b48-b032-314738bb4895","Type":"ContainerStarted","Data":"2fa1314733684f74c0d9fb52e1146cff91961e71e9040fcd3bdaca2c17ae2bfe"} Jul 07 17:23:41.908116 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.908102 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" Jul 07 17:23:41.921989 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.921950 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-f9x6t" podStartSLOduration=1.679833758 podStartE2EDuration="7.92193585s" totalImagesPullingTime="6.242102092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:41.921011569 +0000 UTC m=+87.041215900" watchObservedRunningTime="2026-07-07 17:23:41.92193585 +0000 UTC m=+87.042140228" Jul 07 17:23:41.942775 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.942731 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-844c74b46c-w4gx4" podStartSLOduration=3.69436571 podStartE2EDuration="10.942717966s" totalImagesPullingTime="7.248352256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:41.941525268 +0000 UTC m=+87.061729599" watchObservedRunningTime="2026-07-07 17:23:41.942717966 +0000 UTC m=+87.062922316" Jul 07 17:23:41.968402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.968360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57bcc96f56-6kgf6" podStartSLOduration=3.735556601 podStartE2EDuration="10.968345714s" totalImagesPullingTime="7.232789113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:41.966748831 +0000 UTC m=+87.086953160" watchObservedRunningTime="2026-07-07 17:23:41.968345714 +0000 UTC m=+87.088550045" Jul 07 17:23:41.985562 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:41.985500 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb4d4d4-z9tvx" podStartSLOduration=3.8032814200000002 podStartE2EDuration="10.985474718s" totalImagesPullingTime="7.182193298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:41.984521007 +0000 UTC m=+87.104725348" watchObservedRunningTime="2026-07-07 17:23:41.985474718 +0000 UTC m=+87.105679047" Jul 07 17:23:42.365466 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:42.365379 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:42.444616 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:42.444571 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:42.507154 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:42.507049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:44.872786 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:44.872755 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxjt7" Jul 07 17:23:45.274443 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.274407 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:45.619014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.618917 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:45.720540 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.720510 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:45.820935 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.820897 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:45.929785 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.929747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f85887fb8-kbnnk" event={"ID":"acbc0b73-7f3c-460c-91d5-3b0140aef6c5","Type":"ContainerStarted","Data":"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4"} Jul 07 17:23:45.953409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:45.953362 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f85887fb8-kbnnk" podStartSLOduration=5.573435337 podStartE2EDuration="9.953345455s" totalImagesPullingTime="4.379910118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:45.951967366 +0000 UTC m=+91.072171698" watchObservedRunningTime="2026-07-07 17:23:45.953345455 +0000 UTC m=+91.073549786" Jul 07 17:23:46.798034 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:46.797993 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:46.798236 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:46.798049 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:46.802938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:46.802914 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:46.939577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:46.939537 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:23:47.140877 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.140839 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.241893 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.241851 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.315946 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.315916 2569 patch_prober.go:28] interesting pod/image-registry-67d65479c8-fcb4w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 17:23:47.316113 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.315982 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:23:47.525534 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.525443 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.703599 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.703552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.807954 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.807867 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-42hn9"] Jul 07 17:23:47.814114 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.814079 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.814958 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.814936 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.818178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.817607 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 17:23:47.818178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.817778 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 17:23:47.818178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.817901 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-chml4\"" Jul 07 17:23:47.818178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.818051 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 17:23:47.818178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.818136 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 17:23:47.914266 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.914220 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:47.928513 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928479 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-root\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.928745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-accelerators-collector-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.928745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-textfile\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.928745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-metrics-client-ca\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.928745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.928745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-sys\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.929014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-wtmp\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.929014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-tls\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:47.929014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:47.928876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4r8\" (UniqueName: \"kubernetes.io/projected/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-kube-api-access-7w4r8\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.011030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.010990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.030052 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-textfile\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030052 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030065 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-metrics-client-ca\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-sys\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-wtmp\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-tls\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4r8\" (UniqueName: \"kubernetes.io/projected/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-kube-api-access-7w4r8\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-root\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-accelerators-collector-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.030627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030484 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-textfile\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.031415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030746 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-metrics-client-ca\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.031415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.030965 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-accelerators-collector-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.031415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.031030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-root\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.031415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.031172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-sys\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.031415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.031188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-wtmp\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.033620 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.033580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-tls\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.033726 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.033671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.041026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.041007 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4r8\" (UniqueName: \"kubernetes.io/projected/78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd-kube-api-access-7w4r8\") pod \"node-exporter-42hn9\" (UID: \"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd\") " pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.127396 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.127056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-42hn9" Jul 07 17:23:48.154161 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.154068 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.249336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.249273 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.318012 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.317970 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.657146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.657098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.799578 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.799540 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.884171 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.884059 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:48.995599 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:48.995488 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:50.373638 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:50.373603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:50.468938 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:50.468903 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:50.804071 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:50.804038 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:50.904062 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:50.903980 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.012902 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.012868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.117227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.117064 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.238023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.237995 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:23:51.256375 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.256261 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.367089 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.367048 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.474611 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.474579 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.576112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.576065 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:51.677533 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:51.677387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:52.550755 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:52.550720 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:52.632948 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:52.632850 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:52.836059 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:52.835966 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:52.839949 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:52.839923 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:23:53.053616 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.053579 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:53.139077 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.139042 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:53.777621 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.777590 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-88r7j" Jul 07 17:23:53.789713 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.789681 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:53.885762 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.885725 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:53.956799 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:53.956762 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.035732 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.035657 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.137643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.137599 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.273176 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.273136 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.668506 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:23:54.668481 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d1c5cf_48f4_4b8a_ba0d_88629c0bbefd.slice/crio-640da961dd430928b2e53ab55433771341b1fcab03652af0978d25845b05f6a5 WatchSource:0}: Error finding container 640da961dd430928b2e53ab55433771341b1fcab03652af0978d25845b05f6a5: Status 404 returned error can't find the container with id 640da961dd430928b2e53ab55433771341b1fcab03652af0978d25845b05f6a5 Jul 07 17:23:54.673422 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.672773 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.774940 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.774906 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:54.976356 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:54.976258 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-42hn9" event={"ID":"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd","Type":"ContainerStarted","Data":"640da961dd430928b2e53ab55433771341b1fcab03652af0978d25845b05f6a5"} Jul 07 17:23:55.983163 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:55.983124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zpbpz" event={"ID":"8912dfbf-eba0-43ed-895a-06b623bacc7e","Type":"ContainerStarted","Data":"18d29d1f8985c66887643ea8b29e2f929972e17c9b8e9d8bd2c84b1ea53cbc5e"} Jul 07 17:23:55.983709 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:55.983387 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:56.003911 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:56.003882 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-zpbpz" Jul 07 17:23:56.060939 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:56.054788 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-zpbpz" podStartSLOduration=4.083151226 podStartE2EDuration="25.054771635s" totalImagesPullingTime="20.971620409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:56.023803428 +0000 UTC m=+101.144007759" watchObservedRunningTime="2026-07-07 17:23:56.054771635 +0000 UTC m=+101.174975965" Jul 07 17:23:56.991089 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:56.991047 2569 generic.go:356] "Generic (PLEG): container finished" podID="78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd" containerID="4cb5c784394acd827ee84084f11537004b0c0a52a03aed27d9d839ecc797ab2b" exitCode=0 Jul 07 17:23:56.991560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:56.991092 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-42hn9" event={"ID":"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd","Type":"ContainerDied","Data":"4cb5c784394acd827ee84084f11537004b0c0a52a03aed27d9d839ecc797ab2b"} Jul 07 17:23:57.712268 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:57.712230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:57.836044 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:57.836011 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:58.000223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:58.000143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-42hn9" event={"ID":"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd","Type":"ContainerStarted","Data":"3b2c5faebeaf179ad4e65d4c5e31bf4956a2a480ded924bbfa0988035da2ea0a"} Jul 07 17:23:58.000223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:58.000214 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-42hn9" event={"ID":"78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd","Type":"ContainerStarted","Data":"3a46380cbd83b52efa9c47164885282ed712f011b4c21fb27168844c44aa5aeb"} Jul 07 17:23:58.025718 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:58.025672 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-42hn9" podStartSLOduration=9.54536937 podStartE2EDuration="11.025657967s" totalImagesPullingTime="1.360444398s" totalInitContainerRuntime="119.844199ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:58.024121784 +0000 UTC m=+103.144326117" watchObservedRunningTime="2026-07-07 17:23:58.025657967 +0000 UTC m=+103.145862317" Jul 07 17:23:59.594930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:59.594888 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:23:59.713455 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:59.713415 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:23:59.891622 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:23:59.891590 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:00.008188 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:00.008155 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:01.483812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:01.483769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:01.585451 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:01.585415 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:03.074028 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:03.073993 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:05.363789 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:05.363753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:05.441683 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:05.441652 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:06.861603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:06.861558 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:06.932818 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:06.932786 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:10.057224 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:10.057188 2569 generic.go:356] "Generic (PLEG): container finished" podID="cc9d8bc1-de71-42e4-becc-217e80ac8444" containerID="53571d3cf38d6f9891d227bcb6fe32f3e5d7114522c9a879244d0f57d521b280" exitCode=0 Jul 07 17:24:10.057643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:10.057234 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" event={"ID":"cc9d8bc1-de71-42e4-becc-217e80ac8444","Type":"ContainerDied","Data":"53571d3cf38d6f9891d227bcb6fe32f3e5d7114522c9a879244d0f57d521b280"} Jul 07 17:24:10.057643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:10.057558 2569 scope.go:117] "RemoveContainer" containerID="53571d3cf38d6f9891d227bcb6fe32f3e5d7114522c9a879244d0f57d521b280" Jul 07 17:24:11.062683 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:11.062646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-dk2rt" event={"ID":"cc9d8bc1-de71-42e4-becc-217e80ac8444","Type":"ContainerStarted","Data":"a50eb32a24ecff3d9e951b4a363cb777b32345ef2151fe624579452d39202929"} Jul 07 17:24:11.956844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:11.956810 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:12.059956 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:12.059921 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:12.156905 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:12.156871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:15.868837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:15.868789 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:16.265777 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.265716 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-f85887fb8-kbnnk" podUID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" containerName="console" containerID="cri-o://260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4" gracePeriod=15 Jul 07 17:24:16.542474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.542412 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:24:16.681634 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqfpg\" (UniqueName: \"kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.681823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.681823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681678 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.681823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681730 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.681823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681776 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.681823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.681817 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config\") pod \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\" (UID: \"acbc0b73-7f3c-460c-91d5-3b0140aef6c5\") " Jul 07 17:24:16.682155 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.682125 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config" (OuterVolumeSpecName: "console-config") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:16.682226 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.682152 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:16.682226 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.682161 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:16.684131 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.684096 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:16.684232 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.684151 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg" (OuterVolumeSpecName: "kube-api-access-jqfpg") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "kube-api-access-jqfpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:16.684232 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.684194 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "acbc0b73-7f3c-460c-91d5-3b0140aef6c5" (UID: "acbc0b73-7f3c-460c-91d5-3b0140aef6c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:16.782798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782767 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqfpg\" (UniqueName: \"kubernetes.io/projected/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-kube-api-access-jqfpg\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:16.782798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782794 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-serving-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:16.782798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782803 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-config\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:16.783020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782812 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:16.783020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782821 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-oauth-serving-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:16.783020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:16.782830 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acbc0b73-7f3c-460c-91d5-3b0140aef6c5-console-oauth-config\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:17.088102 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.088068 2569 generic.go:356] "Generic (PLEG): container finished" podID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" containerID="260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4" exitCode=0 Jul 07 17:24:17.088547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.088110 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f85887fb8-kbnnk" event={"ID":"acbc0b73-7f3c-460c-91d5-3b0140aef6c5","Type":"ContainerDied","Data":"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4"} Jul 07 17:24:17.088547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.088114 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f85887fb8-kbnnk" Jul 07 17:24:17.088547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.088143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f85887fb8-kbnnk" event={"ID":"acbc0b73-7f3c-460c-91d5-3b0140aef6c5","Type":"ContainerDied","Data":"0550d48f28cf99ba6f97bfdd462772d920c417375b70174a6af7e0ed0d58de2e"} Jul 07 17:24:17.088547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.088172 2569 scope.go:117] "RemoveContainer" containerID="260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4" Jul 07 17:24:17.097396 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.097374 2569 scope.go:117] "RemoveContainer" containerID="260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4" Jul 07 17:24:17.097661 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:24:17.097642 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4\": container with ID starting with 260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4 not found: ID does not exist" containerID="260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4" Jul 07 17:24:17.097709 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.097671 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4"} err="failed to get container status \"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4\": rpc error: code = NotFound desc = could not find container \"260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4\": container with ID starting with 260c5b7c6821379a6ee362cd159d41c49a1cd5c4ee8c6934418e89b83f505fc4 not found: ID does not exist" Jul 07 17:24:17.111704 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.111683 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:24:17.119203 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.119184 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f85887fb8-kbnnk"] Jul 07 17:24:17.450032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:17.449991 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" path="/var/lib/kubelet/pods/acbc0b73-7f3c-460c-91d5-3b0140aef6c5/volumes" Jul 07 17:24:18.235321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:18.235267 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:20.102897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:20.102863 2569 generic.go:356] "Generic (PLEG): container finished" podID="5876c8dd-c233-475f-a608-85c8f137c6c7" containerID="7c5d9540d8dafa992e4e46898aec4d04d5ce7f551791e2f59b91d298a1b46f1a" exitCode=0 Jul 07 17:24:20.103259 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:20.102942 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" event={"ID":"5876c8dd-c233-475f-a608-85c8f137c6c7","Type":"ContainerDied","Data":"7c5d9540d8dafa992e4e46898aec4d04d5ce7f551791e2f59b91d298a1b46f1a"} Jul 07 17:24:20.103331 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:20.103304 2569 scope.go:117] "RemoveContainer" containerID="7c5d9540d8dafa992e4e46898aec4d04d5ce7f551791e2f59b91d298a1b46f1a" Jul 07 17:24:21.109323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:21.109270 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jwwhp" event={"ID":"5876c8dd-c233-475f-a608-85c8f137c6c7","Type":"ContainerStarted","Data":"7900283a4592e0160f099c2585082bd3fe373ae4abc4037f147ff2372b2a7cbb"} Jul 07 17:24:24.620751 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.620702 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerName="registry" containerID="cri-o://32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a" gracePeriod=30 Jul 07 17:24:24.905872 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.905843 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:24:24.947557 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947531 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947681 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947593 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947681 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947630 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947681 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947700 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947740 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947762 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xph9s\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.947837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.947795 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted\") pod \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\" (UID: \"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50\") " Jul 07 17:24:24.948346 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.948060 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:24.948346 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.948101 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:24.950729 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.950701 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:24.950864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.950839 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s" (OuterVolumeSpecName: "kube-api-access-xph9s") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "kube-api-access-xph9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:24.950939 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.950874 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:24.951010 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.950989 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:24.951068 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.951046 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:24.957605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:24.957583 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" (UID: "4d16c4a0-59f5-44ac-9051-d8edf4e0bb50"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:24:25.049177 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049146 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-trusted-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049177 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049173 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-certificates\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049177 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049184 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-installation-pull-secrets\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049192 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-registry-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049201 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-bound-sa-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049208 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xph9s\" (UniqueName: \"kubernetes.io/projected/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-kube-api-access-xph9s\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049217 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-ca-trust-extracted\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.049409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.049227 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50-image-registry-private-configuration\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:24:25.126780 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.126741 2569 generic.go:356] "Generic (PLEG): container finished" podID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerID="32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a" exitCode=0 Jul 07 17:24:25.126930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.126799 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" Jul 07 17:24:25.126930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.126829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" event={"ID":"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50","Type":"ContainerDied","Data":"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a"} Jul 07 17:24:25.126930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.126901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d65479c8-fcb4w" event={"ID":"4d16c4a0-59f5-44ac-9051-d8edf4e0bb50","Type":"ContainerDied","Data":"a5225d080632de019bbd6abde9c2104503389744259ceda56ebb34b014ba42ec"} Jul 07 17:24:25.126930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.126921 2569 scope.go:117] "RemoveContainer" containerID="32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a" Jul 07 17:24:25.135733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.135712 2569 scope.go:117] "RemoveContainer" containerID="32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a" Jul 07 17:24:25.135998 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:24:25.135979 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a\": container with ID starting with 32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a not found: ID does not exist" containerID="32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a" Jul 07 17:24:25.136056 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.136007 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a"} err="failed to get container status \"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a\": rpc error: code = NotFound desc = could not find container \"32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a\": container with ID starting with 32fbe89581c50bac06e94195732574a56a113487b010eec8a48e784b6e822c9a not found: ID does not exist" Jul 07 17:24:25.151111 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.151083 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:24:25.155801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.155780 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67d65479c8-fcb4w"] Jul 07 17:24:25.450211 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:25.450179 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" path="/var/lib/kubelet/pods/4d16c4a0-59f5-44ac-9051-d8edf4e0bb50/volumes" Jul 07 17:24:35.361208 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:35.361170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:35.450489 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:35.450447 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:35.581679 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:35.581643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:35.649565 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:35.649537 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:40.190014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:40.189977 2569 generic.go:356] "Generic (PLEG): container finished" podID="79a1fb9f-b7f2-4692-9b45-c9871bc7eb58" containerID="c19807e09ac97fc2631ed0d10a3ca29d520aaa70bc7e4f429f18b15f83085e9a" exitCode=0 Jul 07 17:24:40.190439 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:40.190041 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" event={"ID":"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58","Type":"ContainerDied","Data":"c19807e09ac97fc2631ed0d10a3ca29d520aaa70bc7e4f429f18b15f83085e9a"} Jul 07 17:24:40.190439 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:40.190379 2569 scope.go:117] "RemoveContainer" containerID="c19807e09ac97fc2631ed0d10a3ca29d520aaa70bc7e4f429f18b15f83085e9a" Jul 07 17:24:40.655100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:40.655054 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:24:41.196217 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:41.196183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-85ggm" event={"ID":"79a1fb9f-b7f2-4692-9b45-c9871bc7eb58","Type":"ContainerStarted","Data":"8b3322637d26245f1bebcaa9ecc086bbd2b243dcaab3b62464dec9998ce203c1"} Jul 07 17:24:42.966442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:42.966392 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-mtbfz_a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4/cluster-monitoring-operator/0.log" Jul 07 17:24:44.160538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:44.160511 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/init-textfile/0.log" Jul 07 17:24:44.366855 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:44.366824 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/node-exporter/0.log" Jul 07 17:24:44.560269 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:44.560197 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/kube-rbac-proxy/0.log" Jul 07 17:24:48.362947 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:48.362915 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-f9x6t_01e94d75-ad29-4c3d-af53-95489acfc511/prometheus-operator-admission-webhook/0.log" Jul 07 17:24:50.362101 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:50.362070 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-ch6fp_4926c65f-8c54-4aa5-9ea1-b9e59b52b44b/networking-console-plugin/0.log" Jul 07 17:24:50.559614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:50.559589 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:24:50.764471 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:50.764439 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/2.log" Jul 07 17:24:51.165430 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:51.165402 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-zpbpz_8912dfbf-eba0-43ed-895a-06b623bacc7e/download-server/0.log" Jul 07 17:24:51.361117 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:51.361086 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-784748979c-znp9d_bb2371ec-c2d8-4094-a9e4-939d0601b6d5/router/0.log" Jul 07 17:24:51.563392 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:24:51.563294 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqgft_5391d841-23bf-4e93-848e-b2788bc05d76/serve-healthcheck-canary/0.log" Jul 07 17:25:02.100054 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:02.100014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:04.023110 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:04.023074 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:04.097657 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:04.097623 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:08.141672 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:08.141585 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:08.244181 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:08.244143 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:08.350503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:08.350467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:08.859911 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:08.859873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:08.943048 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:08.943009 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:15.985932 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:15.985899 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:16.062724 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:16.062695 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:16.330903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:16.330815 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:19.207931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:19.207895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:19.308153 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:19.308108 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:19.376492 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:19.376458 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:20.364764 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:20.364728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:25:20.463328 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:25:20.463276 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:26:10.827707 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:26:10.827664 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:26:10.919534 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:26:10.919483 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:27:15.311695 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:27:15.311666 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:27:15.315902 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:27:15.315878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:27:15.322552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:27:15.322531 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 17:27:36.080437 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:27:36.080406 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:27:50.653573 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:27:50.653531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:00.621902 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:00.621860 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:00.752441 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:00.752405 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:00.852137 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:00.852104 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:05.359004 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:05.358967 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:05.459531 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:05.459494 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:20.384926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:20.384834 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:28:20.477959 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:28:20.477922 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:29:00.809915 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:29:00.809883 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:29:00.883264 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:29:00.883228 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:29:04.198877 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:29:04.198833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:29:04.314362 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:29:04.314328 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:30:01.666807 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:30:01.666769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:30:56.086169 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:30:56.086129 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:30:56.544036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:30:56.543999 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:00.324602 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:00.324564 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:00.424277 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:00.424246 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:01.814241 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:01.814203 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:02.009401 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:02.009359 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:02.640549 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:02.640509 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:02.881046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:02.881001 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:04.628700 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:04.628667 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:05.268383 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:05.268344 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:06.288098 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:06.288011 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:06.738543 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:06.738514 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:06.858919 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:06.858888 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:10.964812 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:10.964772 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:11.062238 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:11.062202 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:13.354189 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:13.354149 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:14.089632 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:14.089600 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:14.189696 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:14.189636 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:20.701185 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:20.701143 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:20.844835 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:20.844797 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:22.846100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:22.846060 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:35.166762 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:35.166721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:35.286197 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:35.286145 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:37.332635 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:37.332603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:37.444567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:37.444532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:37.737960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:37.737912 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:38.256819 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.256776 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:38.386569 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.386525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:38.496069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496030 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq"] Jul 07 17:31:38.496452 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496432 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerName="registry" Jul 07 17:31:38.496452 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496453 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerName="registry" Jul 07 17:31:38.496668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496468 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" containerName="console" Jul 07 17:31:38.496668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496476 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" containerName="console" Jul 07 17:31:38.496668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496552 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d16c4a0-59f5-44ac-9051-d8edf4e0bb50" containerName="registry" Jul 07 17:31:38.496668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.496567 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbc0b73-7f3c-460c-91d5-3b0140aef6c5" containerName="console" Jul 07 17:31:38.498468 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.498446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.510189 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.510120 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 17:31:38.510189 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.510140 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 17:31:38.510377 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.510128 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 17:31:38.510501 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.510486 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 17:31:38.540412 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.540376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.540588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.540418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.540588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.540436 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.540588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.540490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55xx\" (UniqueName: \"kubernetes.io/projected/402b619f-1f41-4c4b-935d-522b81b6e4b5-kube-api-access-r55xx\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.540588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.540578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.551575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.551547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cpf46\"" Jul 07 17:31:38.552502 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.552480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq"] Jul 07 17:31:38.641179 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.641179 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.641438 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.641438 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55xx\" (UniqueName: \"kubernetes.io/projected/402b619f-1f41-4c4b-935d-522b81b6e4b5-kube-api-access-r55xx\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.641438 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.641559 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.641538 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.643690 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.643652 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/402b619f-1f41-4c4b-935d-522b81b6e4b5-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.643899 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.643879 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.643974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.643900 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/402b619f-1f41-4c4b-935d-522b81b6e4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.657647 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.657620 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55xx\" (UniqueName: \"kubernetes.io/projected/402b619f-1f41-4c4b-935d-522b81b6e4b5-kube-api-access-r55xx\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-k8qxq\" (UID: \"402b619f-1f41-4c4b-935d-522b81b6e4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.805501 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.805418 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:38.808227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.808206 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:38.959258 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.959217 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:38.992873 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:38.992842 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq"] Jul 07 17:31:39.006947 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:31:39.006913 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402b619f_1f41_4c4b_935d_522b81b6e4b5.slice/crio-a8b499293157fee09a5386848bfcd1a2b599e406d67e8c0425536a79dd636d94 WatchSource:0}: Error finding container a8b499293157fee09a5386848bfcd1a2b599e406d67e8c0425536a79dd636d94: Status 404 returned error can't find the container with id a8b499293157fee09a5386848bfcd1a2b599e406d67e8c0425536a79dd636d94 Jul 07 17:31:39.008900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:39.008883 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:31:39.103385 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:39.103277 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:39.217747 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:39.217709 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:39.393559 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:39.393525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:39.852787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:39.852681 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" event={"ID":"402b619f-1f41-4c4b-935d-522b81b6e4b5","Type":"ContainerStarted","Data":"a8b499293157fee09a5386848bfcd1a2b599e406d67e8c0425536a79dd636d94"} Jul 07 17:31:42.871153 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:42.871116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" event={"ID":"402b619f-1f41-4c4b-935d-522b81b6e4b5","Type":"ContainerStarted","Data":"1c38407e67b504243b25679eec333a401e4c3ccbf88f26ce9d3e42393a13fedb"} Jul 07 17:31:43.877053 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:43.877020 2569 generic.go:356] "Generic (PLEG): container finished" podID="402b619f-1f41-4c4b-935d-522b81b6e4b5" containerID="1c38407e67b504243b25679eec333a401e4c3ccbf88f26ce9d3e42393a13fedb" exitCode=0 Jul 07 17:31:43.877503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:43.877114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" event={"ID":"402b619f-1f41-4c4b-935d-522b81b6e4b5","Type":"ContainerDied","Data":"1c38407e67b504243b25679eec333a401e4c3ccbf88f26ce9d3e42393a13fedb"} Jul 07 17:31:44.883441 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:44.883409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" event={"ID":"402b619f-1f41-4c4b-935d-522b81b6e4b5","Type":"ContainerStarted","Data":"6c5032b9c79085ac9eba5d8dbc8b71909436831c0ff5ed6340c2c2c39593c952"} Jul 07 17:31:44.883848 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:44.883543 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:44.911402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:44.911348 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" podStartSLOduration=2.222139132 podStartE2EDuration="6.911332178s" totalImagesPullingTime="3.665950754s" totalInitContainerRuntime="1.023242292s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:31:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:31:44.908712814 +0000 UTC m=+570.028917181" watchObservedRunningTime="2026-07-07 17:31:44.911332178 +0000 UTC m=+570.031536507" Jul 07 17:31:51.645609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:51.645570 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:51.814551 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:51.814514 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:55.890780 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:55.890746 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-k8qxq" Jul 07 17:31:59.768217 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:59.768174 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:31:59.969219 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:31:59.969158 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:00.234064 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:00.234031 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:00.429041 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:00.429006 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:01.534431 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:01.534395 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:01.722540 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:01.722508 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:01.876146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:01.876051 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:03.442365 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:03.442332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:03.701839 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:03.701743 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:04.405409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:04.405370 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:04.628122 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:04.628090 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:06.935019 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:06.934981 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:10.537915 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:10.537877 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:10.678066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:10.678030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:11.845429 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.845396 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w"] Jul 07 17:32:11.848702 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.848685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.854354 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.854325 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 17:32:11.854495 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.854411 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5zp9l\"" Jul 07 17:32:11.898225 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.898190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w"] Jul 07 17:32:11.905638 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.905604 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:11.915563 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fhz\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-kube-api-access-q5fhz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915765 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:11.915853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:11.915841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fhz\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-kube-api-access-q5fhz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016791 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016852 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016852 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016840 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.016929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.016901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.017061 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.017043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.017109 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.017059 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.018907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.018887 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.019048 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.019029 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.033866 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.033834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.045067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.045041 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fhz\" (UniqueName: \"kubernetes.io/projected/4f7a0f87-b8ec-405b-9e62-c84aed63a0cf-kube-api-access-q5fhz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rlb6w\" (UID: \"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.051781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.051749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:12.159157 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.159118 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:12.205614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.205570 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:12.329615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:12.329542 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w"] Jul 07 17:32:12.331963 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:32:12.331935 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7a0f87_b8ec_405b_9e62_c84aed63a0cf.slice/crio-8aadac69a4e74d87d373e97cb7b0db52e5d64117cef2da0b4cef800f238c63c7 WatchSource:0}: Error finding container 8aadac69a4e74d87d373e97cb7b0db52e5d64117cef2da0b4cef800f238c63c7: Status 404 returned error can't find the container with id 8aadac69a4e74d87d373e97cb7b0db52e5d64117cef2da0b4cef800f238c63c7 Jul 07 17:32:13.005466 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.005379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" event={"ID":"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf","Type":"ContainerStarted","Data":"8aadac69a4e74d87d373e97cb7b0db52e5d64117cef2da0b4cef800f238c63c7"} Jul 07 17:32:13.860014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.859968 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn"] Jul 07 17:32:13.863836 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.863805 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:13.874552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.874518 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:32:13.874697 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.874571 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 17:32:13.875277 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.875253 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 17:32:13.875411 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.875302 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 17:32:13.875769 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.875755 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 17:32:13.875852 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.875815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gc8bv\"" Jul 07 17:32:13.933201 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.933145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf2q\" (UniqueName: \"kubernetes.io/projected/1292a3e0-bd1e-4fcb-b923-a88813ab6427-kube-api-access-hzf2q\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:13.933397 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.933219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1292a3e0-bd1e-4fcb-b923-a88813ab6427-manager-config\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:13.933397 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.933278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:13.933397 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.933356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:13.943688 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.943589 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn"] Jul 07 17:32:13.979549 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:13.979512 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:14.034766 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.034637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1292a3e0-bd1e-4fcb-b923-a88813ab6427-manager-config\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.034766 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.034710 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.035352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.034899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.035352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.034944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf2q\" (UniqueName: \"kubernetes.io/projected/1292a3e0-bd1e-4fcb-b923-a88813ab6427-kube-api-access-hzf2q\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.035520 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.035497 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1292a3e0-bd1e-4fcb-b923-a88813ab6427-manager-config\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.038651 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.038602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.038795 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.038685 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1292a3e0-bd1e-4fcb-b923-a88813ab6427-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.060409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.060379 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf2q\" (UniqueName: \"kubernetes.io/projected/1292a3e0-bd1e-4fcb-b923-a88813ab6427-kube-api-access-hzf2q\") pod \"lws-controller-manager-686dcd8c67-5j8wn\" (UID: \"1292a3e0-bd1e-4fcb-b923-a88813ab6427\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.153916 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.153876 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:14.176728 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.176695 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:14.365341 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.365279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn"] Jul 07 17:32:14.366343 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.366270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:14.607606 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.607508 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:14.725067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.725028 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:14.914206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:14.914174 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:15.379215 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:32:15.379146 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1292a3e0_bd1e_4fcb_b923_a88813ab6427.slice/crio-c8ae6e01bdcaadf88a13bf82a2ba04c7590385febb0cfe24588c5b536481322f WatchSource:0}: Error finding container c8ae6e01bdcaadf88a13bf82a2ba04c7590385febb0cfe24588c5b536481322f: Status 404 returned error can't find the container with id c8ae6e01bdcaadf88a13bf82a2ba04c7590385febb0cfe24588c5b536481322f Jul 07 17:32:15.395635 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:15.395609 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:32:15.396803 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:15.396784 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:32:15.444646 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:15.444602 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:32:15.444776 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:15.444698 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:32:15.444776 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:15.444747 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:32:16.026340 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:16.026298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" event={"ID":"1292a3e0-bd1e-4fcb-b923-a88813ab6427","Type":"ContainerStarted","Data":"c8ae6e01bdcaadf88a13bf82a2ba04c7590385febb0cfe24588c5b536481322f"} Jul 07 17:32:16.028278 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:16.028249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" event={"ID":"4f7a0f87-b8ec-405b-9e62-c84aed63a0cf","Type":"ContainerStarted","Data":"b3760cc163796dd2a32eed0fbba62059cf6a22176136bcc77c73cd1cca8bd946"} Jul 07 17:32:16.057687 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:16.057629 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" podStartSLOduration=1.947475031 podStartE2EDuration="5.057613571s" totalImagesPullingTime="3.11013854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:32:16.056891606 +0000 UTC m=+601.177095930" watchObservedRunningTime="2026-07-07 17:32:16.057613571 +0000 UTC m=+601.177817901" Jul 07 17:32:16.159519 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:16.159456 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:16.166210 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:16.166182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:17.033971 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:17.033939 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:17.035159 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:17.035138 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rlb6w" Jul 07 17:32:18.039756 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:18.039716 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" event={"ID":"1292a3e0-bd1e-4fcb-b923-a88813ab6427","Type":"ContainerStarted","Data":"d4a2a96cfbd7e833fc3380846e9ab5b1934fcdcd3f8f31480f7b2f1bb96fe5ad"} Jul 07 17:32:18.040204 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:18.039918 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:18.073336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:18.073268 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" podStartSLOduration=2.826513864 podStartE2EDuration="5.073250866s" totalImagesPullingTime="2.246737002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:32:18.071568605 +0000 UTC m=+603.191772932" watchObservedRunningTime="2026-07-07 17:32:18.073250866 +0000 UTC m=+603.193455196" Jul 07 17:32:28.562941 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:28.562851 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:32:29.050162 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:29.050131 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-5j8wn" Jul 07 17:32:50.171786 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:32:50.171754 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:01.401398 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:01.401355 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:01.529521 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:01.529482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:01.640013 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:01.639978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:31.895979 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:31.895938 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:34.746478 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:34.746434 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:37.124955 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:37.124900 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:38.905241 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:38.905183 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:39.269592 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:39.269500 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:39.586430 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:39.586342 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:41.925961 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:41.925906 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:42.060575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:42.060537 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:42.864643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:42.864610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:43.724032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:43.723994 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:44.969133 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.969096 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv"] Jul 07 17:33:44.972552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.972535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:44.978515 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.978493 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 17:33:44.978671 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.978538 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 17:33:44.987943 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.987922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ff8hg\"" Jul 07 17:33:44.988546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:44.988530 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 17:33:45.002572 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.002540 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv"] Jul 07 17:33:45.006992 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.006967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ppq4\" (UniqueName: \"kubernetes.io/projected/8e9f77d0-856e-46d5-8916-d6d045e2cd4c-kube-api-access-8ppq4\") pod \"dns-operator-controller-manager-864dd78c4c-bszrv\" (UID: \"8e9f77d0-856e-46d5-8916-d6d045e2cd4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:45.107586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.107551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ppq4\" (UniqueName: \"kubernetes.io/projected/8e9f77d0-856e-46d5-8916-d6d045e2cd4c-kube-api-access-8ppq4\") pod \"dns-operator-controller-manager-864dd78c4c-bszrv\" (UID: \"8e9f77d0-856e-46d5-8916-d6d045e2cd4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:45.125537 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.125499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ppq4\" (UniqueName: \"kubernetes.io/projected/8e9f77d0-856e-46d5-8916-d6d045e2cd4c-kube-api-access-8ppq4\") pod \"dns-operator-controller-manager-864dd78c4c-bszrv\" (UID: \"8e9f77d0-856e-46d5-8916-d6d045e2cd4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:45.283418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.283331 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:45.427503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.427480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv"] Jul 07 17:33:45.429534 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:33:45.429507 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9f77d0_856e_46d5_8916_d6d045e2cd4c.slice/crio-13e1d0642ba5a249f090906b33e24b440c90141e7518cf617408ff16cf6c9c5a WatchSource:0}: Error finding container 13e1d0642ba5a249f090906b33e24b440c90141e7518cf617408ff16cf6c9c5a: Status 404 returned error can't find the container with id 13e1d0642ba5a249f090906b33e24b440c90141e7518cf617408ff16cf6c9c5a Jul 07 17:33:45.436579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:45.436490 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" event={"ID":"8e9f77d0-856e-46d5-8916-d6d045e2cd4c","Type":"ContainerStarted","Data":"13e1d0642ba5a249f090906b33e24b440c90141e7518cf617408ff16cf6c9c5a"} Jul 07 17:33:46.827801 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:46.827770 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:46.946149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:46.946110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:48.680715 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:48.680681 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:49.006793 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:49.006750 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:49.457604 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:49.457570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" event={"ID":"8e9f77d0-856e-46d5-8916-d6d045e2cd4c","Type":"ContainerStarted","Data":"c7fb70a04f494e23e4ff82eec95a4429e4757e0435b2c6d27ac33d3463f82f68"} Jul 07 17:33:49.457792 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:49.457703 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:33:49.485388 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:49.485342 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" podStartSLOduration=2.341711147 podStartE2EDuration="5.485325469s" totalImagesPullingTime="3.143614322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:33:49.484015735 +0000 UTC m=+694.604220066" watchObservedRunningTime="2026-07-07 17:33:49.485325469 +0000 UTC m=+694.605529798" Jul 07 17:33:55.464081 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:55.464042 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:33:55.564830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:33:55.564789 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:00.046495 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.046445 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:00.188557 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.188517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:00.463734 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.463706 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bszrv" Jul 07 17:34:00.541014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.540981 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:34:00.544394 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.544374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.548313 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.548277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmq8l\"" Jul 07 17:34:00.567547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.567519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:34:00.642162 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.642129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9696q\" (UniqueName: \"kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.642355 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.642185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.743422 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.743327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.743617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.743437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9696q\" (UniqueName: \"kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.743721 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.743699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.762451 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.762420 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9696q\" (UniqueName: \"kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q\") pod \"kuadrant-operator-controller-manager-86878c769f-j274j\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:00.855474 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:00.855419 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:01.024450 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:01.024422 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:34:01.026898 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:34:01.026866 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493d4ac5_2bf7_4bab_b4ae_88a6c6a81426.slice/crio-8256919970f25d160c630ae3ec9434692a524867327dacc4da556cfa46724240 WatchSource:0}: Error finding container 8256919970f25d160c630ae3ec9434692a524867327dacc4da556cfa46724240: Status 404 returned error can't find the container with id 8256919970f25d160c630ae3ec9434692a524867327dacc4da556cfa46724240 Jul 07 17:34:01.511099 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:01.511058 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" event={"ID":"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426","Type":"ContainerStarted","Data":"8256919970f25d160c630ae3ec9434692a524867327dacc4da556cfa46724240"} Jul 07 17:34:02.349955 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:02.349906 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:02.449415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:02.449383 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:04.736745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:04.736707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:07.541008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:07.540974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" event={"ID":"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426","Type":"ContainerStarted","Data":"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b"} Jul 07 17:34:07.541444 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:07.541127 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:07.571926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:07.571869 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" podStartSLOduration=1.7647981050000001 podStartE2EDuration="7.5718525s" totalImagesPullingTime="5.807054395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:34:07.569316252 +0000 UTC m=+712.689520585" watchObservedRunningTime="2026-07-07 17:34:07.5718525 +0000 UTC m=+712.692056830" Jul 07 17:34:08.100854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:08.100811 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:08.207302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:08.207251 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:14.170399 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:14.170339 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:16.301772 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:16.301730 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:16.397923 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:16.397868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:18.547430 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:18.547400 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:34:20.894974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:20.894935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:20.974483 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:20.974452 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:21.091712 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:21.091671 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:21.188925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:21.188849 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:25.677891 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:25.677855 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:25.742014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:25.741969 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:35.066417 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:35.066383 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:35.206417 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:35.206382 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:39.677030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.676992 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:39.830030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.829987 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:39.831216 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.831196 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:34:39.833692 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.833674 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.837115 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.837093 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 17:34:39.837221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.837208 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d7tsf\"" Jul 07 17:34:39.847940 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.847917 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:34:39.878926 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.878889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4tz\" (UniqueName: \"kubernetes.io/projected/91705533-682f-4b23-9d5e-b7764c12a94a-kube-api-access-wh4tz\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.879069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.878975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91705533-682f-4b23-9d5e-b7764c12a94a-config-file\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.895415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.895387 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:34:39.938246 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.938169 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:39.980380 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.980353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91705533-682f-4b23-9d5e-b7764c12a94a-config-file\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.980509 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.980424 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4tz\" (UniqueName: \"kubernetes.io/projected/91705533-682f-4b23-9d5e-b7764c12a94a-kube-api-access-wh4tz\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.981008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.980988 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91705533-682f-4b23-9d5e-b7764c12a94a-config-file\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:39.992223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:39.992192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4tz\" (UniqueName: \"kubernetes.io/projected/91705533-682f-4b23-9d5e-b7764c12a94a-kube-api-access-wh4tz\") pod \"limitador-limitador-54d6d988d5-8ftkv\" (UID: \"91705533-682f-4b23-9d5e-b7764c12a94a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:40.144073 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:40.144037 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:40.288844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:40.288809 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:34:40.291841 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:34:40.291807 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91705533_682f_4b23_9d5e_b7764c12a94a.slice/crio-a94043e2a1815b813a3bf408d2b223160b98021f7a2d57161418d75d7e7d4ff7 WatchSource:0}: Error finding container a94043e2a1815b813a3bf408d2b223160b98021f7a2d57161418d75d7e7d4ff7: Status 404 returned error can't find the container with id a94043e2a1815b813a3bf408d2b223160b98021f7a2d57161418d75d7e7d4ff7 Jul 07 17:34:40.692392 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:40.692353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" event={"ID":"91705533-682f-4b23-9d5e-b7764c12a94a","Type":"ContainerStarted","Data":"a94043e2a1815b813a3bf408d2b223160b98021f7a2d57161418d75d7e7d4ff7"} Jul 07 17:34:40.692982 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:40.692952 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:40.821943 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:40.821861 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:34:43.708799 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:43.708761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" event={"ID":"91705533-682f-4b23-9d5e-b7764c12a94a","Type":"ContainerStarted","Data":"ff073f841ae53502757a5ed744150a860ebc7b289c10621357b0ee956d33267b"} Jul 07 17:34:43.709328 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:43.708909 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:34:43.729011 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:43.728953 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" podStartSLOduration=2.115425867 podStartE2EDuration="4.728934123s" totalImagesPullingTime="2.613508256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:34:43.728655807 +0000 UTC m=+748.848860136" watchObservedRunningTime="2026-07-07 17:34:43.728934123 +0000 UTC m=+748.849138456" Jul 07 17:34:54.714973 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:34:54.714941 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8ftkv" Jul 07 17:35:01.266798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:01.266753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:14.023081 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:14.023046 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:14.197020 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:14.196984 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:20.800838 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:20.800798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:20.897255 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:20.897220 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:20.996155 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:20.996120 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:21.097745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:21.097646 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:23.513568 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:23.513538 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:23.625768 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:23.625734 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:51.202076 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:51.202032 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:53.443179 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:53.443139 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:56.043394 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:56.043357 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:56.558233 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:56.558191 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:56.693461 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:56.693422 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:57.075682 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:57.075651 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:57.178519 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:57.178467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:35:57.250638 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:35:57.250603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:36:07.057642 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:07.057604 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:36:07.152710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:07.152678 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:36:19.328026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:19.327990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:36:19.852076 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:19.852046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:36:34.414047 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:34.414008 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:36:35.279971 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:36:35.279939 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:37:02.517399 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.517363 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr"] Jul 07 17:37:02.517960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.517929 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:02.521052 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.521036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.533070 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.533038 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 17:37:02.533240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.533121 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 17:37:02.533240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.533045 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bl7jx\"" Jul 07 17:37:02.533240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.533122 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 17:37:02.542991 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.542964 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr"] Jul 07 17:37:02.569928 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.569895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f72135d-0228-48ad-8c89-240284ab53b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.570072 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.569950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.570072 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.570011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8l5\" (UniqueName: \"kubernetes.io/projected/9f72135d-0228-48ad-8c89-240284ab53b9-kube-api-access-sl8l5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.570305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.570088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.570305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.570123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.570305 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.570143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f72135d-0228-48ad-8c89-240284ab53b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8l5\" (UniqueName: \"kubernetes.io/projected/9f72135d-0228-48ad-8c89-240284ab53b9-kube-api-access-sl8l5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671549 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671720 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.671872 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.671772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.673673 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.673656 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f72135d-0228-48ad-8c89-240284ab53b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.673846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.673828 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f72135d-0228-48ad-8c89-240284ab53b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.689822 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.689792 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8l5\" (UniqueName: \"kubernetes.io/projected/9f72135d-0228-48ad-8c89-240284ab53b9-kube-api-access-sl8l5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xw9sr\" (UID: \"9f72135d-0228-48ad-8c89-240284ab53b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:02.855272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:02.855178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:03.006610 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:03.006582 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr"] Jul 07 17:37:03.008182 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:37:03.008157 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f72135d_0228_48ad_8c89_240284ab53b9.slice/crio-f37c3aba1ba039d3a4370c3f65488caea1d72db8691b85ddd5383ea9e9067398 WatchSource:0}: Error finding container f37c3aba1ba039d3a4370c3f65488caea1d72db8691b85ddd5383ea9e9067398: Status 404 returned error can't find the container with id f37c3aba1ba039d3a4370c3f65488caea1d72db8691b85ddd5383ea9e9067398 Jul 07 17:37:03.009573 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:03.009557 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:37:03.351652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:03.351615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" event={"ID":"9f72135d-0228-48ad-8c89-240284ab53b9","Type":"ContainerStarted","Data":"f37c3aba1ba039d3a4370c3f65488caea1d72db8691b85ddd5383ea9e9067398"} Jul 07 17:37:04.111689 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:04.111639 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:37:04.357895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:04.357853 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" event={"ID":"9f72135d-0228-48ad-8c89-240284ab53b9","Type":"ContainerStarted","Data":"0464584a3405e6db0c275f4afbab5fecffd8243f72774078d93080b9ae66288b"} Jul 07 17:37:04.391179 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:04.391131 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" podStartSLOduration=1.187348424 podStartE2EDuration="2.391114593s" totalImagesPullingTime="1.203766169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:37:04.38881833 +0000 UTC m=+889.509022661" watchObservedRunningTime="2026-07-07 17:37:04.391114593 +0000 UTC m=+889.511318922" Jul 07 17:37:08.548025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:08.547990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:12.855624 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:12.855585 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:12.856027 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:12.855640 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:12.868922 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:12.868894 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:13.410684 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:13.410648 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xw9sr" Jul 07 17:37:15.428311 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:15.428265 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:37:15.431329 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:15.431277 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:37:28.202391 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:28.202352 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:29.667550 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:29.667499 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:37:48.880350 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.880314 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq"] Jul 07 17:37:48.882831 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.882809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.885535 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.885514 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 17:37:48.896112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.896087 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq"] Jul 07 17:37:48.928969 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.928933 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:48.970483 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970448 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fced498-f4db-46dd-a055-26fdab2b9d37-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.970685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.970685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.970685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.970685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:48.970685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:48.970642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmk28\" (UniqueName: \"kubernetes.io/projected/4fced498-f4db-46dd-a055-26fdab2b9d37-kube-api-access-fmk28\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fced498-f4db-46dd-a055-26fdab2b9d37-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.071925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.071762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmk28\" (UniqueName: \"kubernetes.io/projected/4fced498-f4db-46dd-a055-26fdab2b9d37-kube-api-access-fmk28\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.072142 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.072114 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.072196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.072147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.072247 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.072196 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.074061 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.074040 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fced498-f4db-46dd-a055-26fdab2b9d37-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.074244 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.074227 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fced498-f4db-46dd-a055-26fdab2b9d37-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.081947 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.081926 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmk28\" (UniqueName: \"kubernetes.io/projected/4fced498-f4db-46dd-a055-26fdab2b9d37-kube-api-access-fmk28\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s7xrq\" (UID: \"4fced498-f4db-46dd-a055-26fdab2b9d37\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.196403 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.196357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:49.343785 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.343759 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq"] Jul 07 17:37:49.345862 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:37:49.345834 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fced498_f4db_46dd_a055_26fdab2b9d37.slice/crio-2631b7695610b48c12ed05b3bb4c122dea99ef152cf6895e5a0aceaebbce6cf5 WatchSource:0}: Error finding container 2631b7695610b48c12ed05b3bb4c122dea99ef152cf6895e5a0aceaebbce6cf5: Status 404 returned error can't find the container with id 2631b7695610b48c12ed05b3bb4c122dea99ef152cf6895e5a0aceaebbce6cf5 Jul 07 17:37:49.569568 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:49.569539 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" event={"ID":"4fced498-f4db-46dd-a055-26fdab2b9d37","Type":"ContainerStarted","Data":"2631b7695610b48c12ed05b3bb4c122dea99ef152cf6895e5a0aceaebbce6cf5"} Jul 07 17:37:50.575733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:50.575696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" event={"ID":"4fced498-f4db-46dd-a055-26fdab2b9d37","Type":"ContainerStarted","Data":"bc357543696b225ebbc28482efcc5dd03b9b0f1b70ed8404b69a485fc570d4bc"} Jul 07 17:37:50.598371 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:50.598309 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" podStartSLOduration=2.40949621 podStartE2EDuration="2.598258922s" totalImagesPullingTime="188.762712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:37:50.595767612 +0000 UTC m=+935.715971942" watchObservedRunningTime="2026-07-07 17:37:50.598258922 +0000 UTC m=+935.718463254" Jul 07 17:37:50.863823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:50.863732 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:37:54.864116 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:54.864059 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:56.042026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:56.041988 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:37:59.197240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:59.197201 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:59.197775 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:59.197274 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:59.210200 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:59.210166 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:37:59.627831 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:37:59.627744 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s7xrq" Jul 07 17:38:08.971872 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:08.971839 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt"] Jul 07 17:38:08.975415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:08.975392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:08.978312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:08.978277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 17:38:08.984661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:08.984636 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt"] Jul 07 17:38:09.018560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.018526 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:38:09.052345 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.052504 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxdh\" (UniqueName: \"kubernetes.io/projected/f386777a-7c44-4e75-8c21-cdd70d7ec66e-kube-api-access-dlxdh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.052504 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052416 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.052504 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.052627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052515 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.052627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.052609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.153807 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxdh\" (UniqueName: \"kubernetes.io/projected/f386777a-7c44-4e75-8c21-cdd70d7ec66e-kube-api-access-dlxdh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.153937 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154325 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.154269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154385 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.154362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.154432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.154395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.156410 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.156383 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f386777a-7c44-4e75-8c21-cdd70d7ec66e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.156691 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.156669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386777a-7c44-4e75-8c21-cdd70d7ec66e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.165814 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.165790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxdh\" (UniqueName: \"kubernetes.io/projected/f386777a-7c44-4e75-8c21-cdd70d7ec66e-kube-api-access-dlxdh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt\" (UID: \"f386777a-7c44-4e75-8c21-cdd70d7ec66e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.288719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.288629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:09.435112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.435082 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt"] Jul 07 17:38:09.436676 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:38:09.436651 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf386777a_7c44_4e75_8c21_cdd70d7ec66e.slice/crio-52e0faebce77dc267282acf011660202beaed3a7952eb7247766f6894ada7ef3 WatchSource:0}: Error finding container 52e0faebce77dc267282acf011660202beaed3a7952eb7247766f6894ada7ef3: Status 404 returned error can't find the container with id 52e0faebce77dc267282acf011660202beaed3a7952eb7247766f6894ada7ef3 Jul 07 17:38:09.664878 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:09.664844 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" event={"ID":"f386777a-7c44-4e75-8c21-cdd70d7ec66e","Type":"ContainerStarted","Data":"52e0faebce77dc267282acf011660202beaed3a7952eb7247766f6894ada7ef3"} Jul 07 17:38:10.671571 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:10.671533 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" event={"ID":"f386777a-7c44-4e75-8c21-cdd70d7ec66e","Type":"ContainerStarted","Data":"64b9803be43df035061f8da543479b17e3559179cdd2645b7b88ac832af45184"} Jul 07 17:38:10.693027 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:10.692975 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" podStartSLOduration=2.523143666 podStartE2EDuration="2.692959101s" totalImagesPullingTime="169.815435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:38:10.692201245 +0000 UTC m=+955.812405574" watchObservedRunningTime="2026-07-07 17:38:10.692959101 +0000 UTC m=+955.813163458" Jul 07 17:38:11.752573 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:11.752539 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:38:19.289787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:19.289739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:19.290240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:19.289807 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:19.302679 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:19.302651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:38:19.726208 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:38:19.726174 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt" Jul 07 17:39:04.059355 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:04.059315 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:39:04.161408 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:04.161370 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:39:05.258930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:05.258895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:39:05.356830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:05.356797 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:39:33.865947 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:33.865910 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:39:43.559299 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:43.559254 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:39:54.524557 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:39:54.524520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:40:03.658634 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:03.658601 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:40:14.857610 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:14.857525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:40:16.836717 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:16.836667 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:18.133726 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:18.133697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:22.526965 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:22.526926 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:54.939140 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:54.939099 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:55.039445 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:55.039398 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:55.191601 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:55.191476 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:56.133566 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:40:56.133527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:01.573846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:01.573813 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:01.677010 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:01.676976 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:01.773455 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:01.773425 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:01.954560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:01.954527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:07.866343 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:07.866305 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:07.966911 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:07.966863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:25.060969 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:25.060925 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:41:34.364759 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:34.364722 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:41:43.261676 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:43.261639 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:41:54.552848 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:41:54.552810 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:42:05.486541 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:05.486504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:42:15.465066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:15.465038 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:42:15.468233 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:15.468212 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:42:16.359723 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:16.359687 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:42:26.855438 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:26.855400 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:42:36.859521 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:36.859473 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:42:59.256007 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:42:59.255968 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:43:10.854592 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:43:10.854513 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:43:19.144469 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:43:19.144431 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:43:20.450137 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:43:20.450103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:43:35.461892 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:43:35.461856 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:43:49.365688 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:43:49.365651 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:44:01.366488 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:44:01.366453 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:44:10.956441 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:44:10.956402 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:44:27.165584 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:44:27.165543 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:45:08.125143 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:08.125107 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:45:08.125702 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:08.125330 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" podUID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" containerName="manager" containerID="cri-o://4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b" gracePeriod=10 Jul 07 17:45:11.582338 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.582305 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:45:11.684809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.684723 2569 generic.go:356] "Generic (PLEG): container finished" podID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" containerID="4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b" exitCode=0 Jul 07 17:45:11.684809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.684766 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" Jul 07 17:45:11.684988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.684812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" event={"ID":"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426","Type":"ContainerDied","Data":"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b"} Jul 07 17:45:11.684988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.684863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j" event={"ID":"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426","Type":"ContainerDied","Data":"8256919970f25d160c630ae3ec9434692a524867327dacc4da556cfa46724240"} Jul 07 17:45:11.684988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.684886 2569 scope.go:117] "RemoveContainer" containerID="4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b" Jul 07 17:45:11.694460 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.694443 2569 scope.go:117] "RemoveContainer" containerID="4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b" Jul 07 17:45:11.694708 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:45:11.694689 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b\": container with ID starting with 4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b not found: ID does not exist" containerID="4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b" Jul 07 17:45:11.694751 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.694719 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b"} err="failed to get container status \"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b\": rpc error: code = NotFound desc = could not find container \"4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b\": container with ID starting with 4f7362a0d4d35d98efe88837f6ffcbdee3d03902ab1b0cc91b01ff99edfe979b not found: ID does not exist" Jul 07 17:45:11.724014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.723992 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume\") pod \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " Jul 07 17:45:11.724131 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.724093 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9696q\" (UniqueName: \"kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q\") pod \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\" (UID: \"493d4ac5-2bf7-4bab-b4ae-88a6c6a81426\") " Jul 07 17:45:11.724372 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.724351 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" (UID: "493d4ac5-2bf7-4bab-b4ae-88a6c6a81426"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:45:11.726338 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.726312 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q" (OuterVolumeSpecName: "kube-api-access-9696q") pod "493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" (UID: "493d4ac5-2bf7-4bab-b4ae-88a6c6a81426"). InnerVolumeSpecName "kube-api-access-9696q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:45:11.825373 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.825328 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9696q\" (UniqueName: \"kubernetes.io/projected/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-kube-api-access-9696q\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:45:11.825373 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:11.825365 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426-extensions-socket-volume\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:45:12.010549 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:12.010513 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:45:12.016009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:12.015984 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j274j"] Jul 07 17:45:13.451489 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:13.451457 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" path="/var/lib/kubelet/pods/493d4ac5-2bf7-4bab-b4ae-88a6c6a81426/volumes" Jul 07 17:45:49.749175 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:45:49.749136 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:46:29.093636 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:46:29.093594 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:46:41.092865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:46:41.092828 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:46:49.497430 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:46:49.497395 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:02.332823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:02.332788 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:13.887698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:13.887663 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:15.492067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:15.492035 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:47:15.495585 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:15.495559 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:47:24.503829 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:24.503790 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:33.321922 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:33.321886 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:44.208816 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:44.208776 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:47:55.817264 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:47:55.817222 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:48:05.115539 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:05.115493 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:48:09.255867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:09.255830 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:48:09.362685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:09.362639 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:48:17.495212 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:17.495176 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:48:42.831907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:42.831870 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:48:50.315501 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:48:50.315463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:49:01.982977 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:01.982943 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:03.108037 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:03.108004 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:49:15.180717 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:15.180629 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:49:17.126449 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.126417 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ddfbbf57d-7zhwz"] Jul 07 17:49:17.126847 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.126784 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" containerName="manager" Jul 07 17:49:17.126847 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.126796 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" containerName="manager" Jul 07 17:49:17.126952 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.126878 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="493d4ac5-2bf7-4bab-b4ae-88a6c6a81426" containerName="manager" Jul 07 17:49:17.129940 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.129920 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.140207 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.140186 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 17:49:17.140563 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.140547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 17:49:17.141564 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.141546 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 17:49:17.141666 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.141589 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zm2j4\"" Jul 07 17:49:17.141666 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.141627 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 17:49:17.142001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.141666 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 17:49:17.192800 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.192763 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 17:49:17.213967 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.213941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ddfbbf57d-7zhwz"] Jul 07 17:49:17.218004 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.217972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-service-ca\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-trusted-ca-bundle\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218149 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bzs\" (UniqueName: \"kubernetes.io/projected/6bad37ed-24d3-4542-9f25-c90237db54cb-kube-api-access-v7bzs\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-console-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-oauth-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.218424 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.218333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-oauth-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.290087 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.290052 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:17.319183 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-oauth-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-service-ca\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-trusted-ca-bundle\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bzs\" (UniqueName: \"kubernetes.io/projected/6bad37ed-24d3-4542-9f25-c90237db54cb-kube-api-access-v7bzs\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-console-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.319754 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.319415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-oauth-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.320101 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.320073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-oauth-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.320200 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.320097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-service-ca\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.320252 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.320229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-trusted-ca-bundle\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.320321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.320177 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bad37ed-24d3-4542-9f25-c90237db54cb-console-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.321895 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.321872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-serving-cert\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.321990 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.321895 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bad37ed-24d3-4542-9f25-c90237db54cb-console-oauth-config\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.331949 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.331923 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bzs\" (UniqueName: \"kubernetes.io/projected/6bad37ed-24d3-4542-9f25-c90237db54cb-kube-api-access-v7bzs\") pod \"console-6ddfbbf57d-7zhwz\" (UID: \"6bad37ed-24d3-4542-9f25-c90237db54cb\") " pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.375941 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.375896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:17.439822 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.439776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:17.592529 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.592496 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ddfbbf57d-7zhwz"] Jul 07 17:49:17.596018 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:49:17.595991 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bad37ed_24d3_4542_9f25_c90237db54cb.slice/crio-4437cab9ec046283950c985fe0c0602fed85f0167b6d5b63c05bf41a86a91477 WatchSource:0}: Error finding container 4437cab9ec046283950c985fe0c0602fed85f0167b6d5b63c05bf41a86a91477: Status 404 returned error can't find the container with id 4437cab9ec046283950c985fe0c0602fed85f0167b6d5b63c05bf41a86a91477 Jul 07 17:49:17.832441 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.832339 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddfbbf57d-7zhwz" event={"ID":"6bad37ed-24d3-4542-9f25-c90237db54cb","Type":"ContainerStarted","Data":"705fb442c61c958e06c5d2284d1eab01be68d22cc41683287378b3be08446b40"} Jul 07 17:49:17.832441 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.832402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddfbbf57d-7zhwz" event={"ID":"6bad37ed-24d3-4542-9f25-c90237db54cb","Type":"ContainerStarted","Data":"4437cab9ec046283950c985fe0c0602fed85f0167b6d5b63c05bf41a86a91477"} Jul 07 17:49:17.855662 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:17.855608 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ddfbbf57d-7zhwz" podStartSLOduration=0.855593292 podStartE2EDuration="855.593292ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:49:17.853152142 +0000 UTC m=+1622.973356471" watchObservedRunningTime="2026-07-07 17:49:17.855593292 +0000 UTC m=+1622.975797622" Jul 07 17:49:27.440939 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:27.440894 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:27.441464 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:27.440995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:27.451128 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:27.451104 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:27.883624 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:27.883596 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ddfbbf57d-7zhwz" Jul 07 17:49:28.055215 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:28.055180 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:28.165580 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:28.165545 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:28.234700 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:28.234654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:49:33.087767 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:33.087725 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:49:42.982965 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:42.982930 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:49:52.388352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:49:52.388314 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:02.691700 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:02.691662 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:12.182256 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:12.182218 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:22.079771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:22.079733 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:35.279673 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:35.279643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:44.384799 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:44.384762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:50:54.285447 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:50:54.285411 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:51:03.979018 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:51:03.978977 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:51:16.484896 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:51:16.484859 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:51:25.589877 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:51:25.589824 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:51:38.386837 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:51:38.386801 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:51:51.092603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:51:51.092566 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:00.787672 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:00.787631 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:03.467045 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:03.466990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:10.994134 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:10.994015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:14.686265 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:14.686223 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:15.521082 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:15.521057 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:52:15.524725 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:15.524702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:52:19.331239 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:19.331203 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:23.480107 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:23.480071 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:24.497720 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:24.497689 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:27.671860 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:27.671826 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:27.778890 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:27.778855 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:28.146084 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:28.146056 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:33.295660 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:33.295615 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:34.346915 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.346878 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:34.472706 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.472670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:34.540191 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.540161 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:34.543663 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.543639 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.551845 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.551820 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b0a4cbb-openshift-default-dockercfg-zsvk8\"" Jul 07 17:52:34.578748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.578715 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:34.585653 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.585629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:34.600499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600644 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600644 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwn2\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.600840 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.600769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.652628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.652590 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:34.701744 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701744 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701859 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwn2\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.701980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702400 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.701984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702400 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.702245 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702400 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.702312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702544 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.702399 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702544 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.702501 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.702914 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.702889 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.704423 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.704404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.704532 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.704521 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.711153 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.711126 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.711313 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.711202 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwn2\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2\") pod \"e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:34.855858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:34.855761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:35.002131 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.002106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:35.004407 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:52:35.004369 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b79ff4b_9414_42ab_b1b9_8a3070685d34.slice/crio-f32a690d460470ff8cbe62151dcf1e05668263e8312f1e25e60f58957ec5af5d WatchSource:0}: Error finding container f32a690d460470ff8cbe62151dcf1e05668263e8312f1e25e60f58957ec5af5d: Status 404 returned error can't find the container with id f32a690d460470ff8cbe62151dcf1e05668263e8312f1e25e60f58957ec5af5d Jul 07 17:52:35.006497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.006457 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:35.006573 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.006531 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:35.006573 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.006563 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:35.760667 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.760621 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" event={"ID":"5b79ff4b-9414-42ab-b1b9-8a3070685d34","Type":"ContainerStarted","Data":"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051"} Jul 07 17:52:35.761058 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.760683 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" event={"ID":"5b79ff4b-9414-42ab-b1b9-8a3070685d34","Type":"ContainerStarted","Data":"f32a690d460470ff8cbe62151dcf1e05668263e8312f1e25e60f58957ec5af5d"} Jul 07 17:52:35.768580 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.768545 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:35.792225 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.792178 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" podStartSLOduration=1.792161681 podStartE2EDuration="1.792161681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:35.790734308 +0000 UTC m=+1820.910938637" watchObservedRunningTime="2026-07-07 17:52:35.792161681 +0000 UTC m=+1820.912366011" Jul 07 17:52:35.856040 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.856009 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:35.868056 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.868020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:35.872632 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.872606 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:35.876014 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.875996 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:35.879001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.878979 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5b0a4cbb-serving-cert\"" Jul 07 17:52:35.879001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.878996 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:52:35.890412 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.890387 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:35.941876 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:35.941820 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:36.014809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.014710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.014809 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.014770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.015052 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.014840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6db\" (UniqueName: \"kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.116128 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.116088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.116361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.116152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.116361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.116191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6db\" (UniqueName: \"kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.116361 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:52:36.116248 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-5b0a4cbb-serving-cert: secret "maas-api-e2e-ait-5b0a4cbb-serving-cert" not found Jul 07 17:52:36.116361 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:52:36.116347 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls podName:b31348c6-c62b-4560-b3a8-6ce6ecec8278 nodeName:}" failed. No retries permitted until 2026-07-07 17:52:36.616328608 +0000 UTC m=+1821.736532916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls") pod "maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" (UID: "b31348c6-c62b-4560-b3a8-6ce6ecec8278") : secret "maas-api-e2e-ait-5b0a4cbb-serving-cert" not found Jul 07 17:52:36.116827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.116806 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.130736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.130713 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6db\" (UniqueName: \"kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.456056 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.456018 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:36.549382 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.549342 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:36.620908 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.620855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.623527 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.623494 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") pod \"maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.787670 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.787578 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:36.861738 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.861711 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:36.941648 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.941561 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:36.943973 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:52:36.943947 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31348c6_c62b_4560_b3a8_6ce6ecec8278.slice/crio-b1ac9cbc655c870b80be68d0cd86c07925e9f42144c7513ceaff498869febb98 WatchSource:0}: Error finding container b1ac9cbc655c870b80be68d0cd86c07925e9f42144c7513ceaff498869febb98: Status 404 returned error can't find the container with id b1ac9cbc655c870b80be68d0cd86c07925e9f42144c7513ceaff498869febb98 Jul 07 17:52:36.945601 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:36.945580 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:52:37.771842 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:37.771804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" event={"ID":"b31348c6-c62b-4560-b3a8-6ce6ecec8278","Type":"ContainerStarted","Data":"b1ac9cbc655c870b80be68d0cd86c07925e9f42144c7513ceaff498869febb98"} Jul 07 17:52:37.772214 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:37.772191 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:37.773417 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:37.773392 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:38.524392 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:38.524354 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:39.784394 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:39.784358 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" event={"ID":"b31348c6-c62b-4560-b3a8-6ce6ecec8278","Type":"ContainerStarted","Data":"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e"} Jul 07 17:52:39.784891 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:39.784518 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:39.813627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:39.807256 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" podStartSLOduration=2.205006335 podStartE2EDuration="4.807235145s" totalImagesPullingTime="2.60222881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:39.806807864 +0000 UTC m=+1824.927012195" watchObservedRunningTime="2026-07-07 17:52:39.807235145 +0000 UTC m=+1824.927439473" Jul 07 17:52:41.011969 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.011926 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:41.167980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.167937 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:41.168189 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.168163 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" podUID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" containerName="istio-proxy" containerID="cri-o://b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051" gracePeriod=30 Jul 07 17:52:41.183711 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.183679 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:41.311207 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.311014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:41.454562 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.454530 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:41.480754 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.480719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:41.793966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:41.793919 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" podUID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" containerName="maas-api" containerID="cri-o://b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e" gracePeriod=30 Jul 07 17:52:42.076855 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.076828 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:42.177457 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.177420 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") pod \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " Jul 07 17:52:42.177657 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.177471 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg6db\" (UniqueName: \"kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db\") pod \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " Jul 07 17:52:42.177657 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.177606 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca\") pod \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\" (UID: \"b31348c6-c62b-4560-b3a8-6ce6ecec8278\") " Jul 07 17:52:42.177959 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.177921 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b31348c6-c62b-4560-b3a8-6ce6ecec8278" (UID: "b31348c6-c62b-4560-b3a8-6ce6ecec8278"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:52:42.179878 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.179855 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b31348c6-c62b-4560-b3a8-6ce6ecec8278" (UID: "b31348c6-c62b-4560-b3a8-6ce6ecec8278"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:52:42.179966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.179909 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db" (OuterVolumeSpecName: "kube-api-access-jg6db") pod "b31348c6-c62b-4560-b3a8-6ce6ecec8278" (UID: "b31348c6-c62b-4560-b3a8-6ce6ecec8278"). InnerVolumeSpecName "kube-api-access-jg6db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:52:42.278795 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.278755 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31348c6-c62b-4560-b3a8-6ce6ecec8278-openshift-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:42.278795 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.278789 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31348c6-c62b-4560-b3a8-6ce6ecec8278-maas-api-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:42.278795 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.278800 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6db\" (UniqueName: \"kubernetes.io/projected/b31348c6-c62b-4560-b3a8-6ce6ecec8278-kube-api-access-jg6db\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:42.801707 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.801661 2569 generic.go:356] "Generic (PLEG): container finished" podID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" containerID="b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e" exitCode=0 Jul 07 17:52:42.801930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.801799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" event={"ID":"b31348c6-c62b-4560-b3a8-6ce6ecec8278","Type":"ContainerDied","Data":"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e"} Jul 07 17:52:42.801930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.801852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" event={"ID":"b31348c6-c62b-4560-b3a8-6ce6ecec8278","Type":"ContainerDied","Data":"b1ac9cbc655c870b80be68d0cd86c07925e9f42144c7513ceaff498869febb98"} Jul 07 17:52:42.801930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.801883 2569 scope.go:117] "RemoveContainer" containerID="b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e" Jul 07 17:52:42.802120 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.802053 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j" Jul 07 17:52:42.830667 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.830645 2569 scope.go:117] "RemoveContainer" containerID="b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e" Jul 07 17:52:42.830984 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:52:42.830963 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e\": container with ID starting with b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e not found: ID does not exist" containerID="b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e" Jul 07 17:52:42.831218 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.830996 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e"} err="failed to get container status \"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e\": rpc error: code = NotFound desc = could not find container \"b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e\": container with ID starting with b5639d36fca3d9ccbd53ae3d57805e0443b87fc64d876b85c33b57d32801394e not found: ID does not exist" Jul 07 17:52:42.840854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.840830 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:42.845358 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:42.845330 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5b0a4cbb-7b8dd8cddb-f5k6j"] Jul 07 17:52:43.451620 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:43.451584 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" path="/var/lib/kubelet/pods/b31348c6-c62b-4560-b3a8-6ce6ecec8278/volumes" Jul 07 17:52:44.388046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:44.388004 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:46.631909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.631884 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:46.720601 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720511 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqwn2\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720601 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720557 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720601 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720592 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720627 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720708 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720781 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720808 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.720854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720839 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token\") pod \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\" (UID: \"5b79ff4b-9414-42ab-b1b9-8a3070685d34\") " Jul 07 17:52:46.721164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720936 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:52:46.721164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.720965 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:52:46.721164 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721024 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:52:46.721442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721166 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:52:46.721442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721226 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.721442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721249 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.721442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721264 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.721821 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.721799 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:52:46.723116 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.723086 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:52:46.723237 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.723201 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:52:46.723307 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.723235 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:52:46.723307 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.723300 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2" (OuterVolumeSpecName: "kube-api-access-kqwn2") pod "5b79ff4b-9414-42ab-b1b9-8a3070685d34" (UID: "5b79ff4b-9414-42ab-b1b9-8a3070685d34"). InnerVolumeSpecName "kube-api-access-kqwn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:52:46.821853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821822 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.821853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821848 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.821853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821857 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.822100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821866 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwn2\" (UniqueName: \"kubernetes.io/projected/5b79ff4b-9414-42ab-b1b9-8a3070685d34-kube-api-access-kqwn2\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.822100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821874 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.822100 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.821884 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b79ff4b-9414-42ab-b1b9-8a3070685d34-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:52:46.824892 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.824862 2569 generic.go:356] "Generic (PLEG): container finished" podID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" containerID="b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051" exitCode=0 Jul 07 17:52:46.825043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.824912 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" Jul 07 17:52:46.825043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.824948 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" event={"ID":"5b79ff4b-9414-42ab-b1b9-8a3070685d34","Type":"ContainerDied","Data":"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051"} Jul 07 17:52:46.825043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.825004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph" event={"ID":"5b79ff4b-9414-42ab-b1b9-8a3070685d34","Type":"ContainerDied","Data":"f32a690d460470ff8cbe62151dcf1e05668263e8312f1e25e60f58957ec5af5d"} Jul 07 17:52:46.825043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.825026 2569 scope.go:117] "RemoveContainer" containerID="b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051" Jul 07 17:52:46.835013 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.834998 2569 scope.go:117] "RemoveContainer" containerID="b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051" Jul 07 17:52:46.835262 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:52:46.835243 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051\": container with ID starting with b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051 not found: ID does not exist" containerID="b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051" Jul 07 17:52:46.835352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.835270 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051"} err="failed to get container status \"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051\": rpc error: code = NotFound desc = could not find container \"b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051\": container with ID starting with b9f327004535144d7d7e50d3f9cb96de653af86f923d6c977b4b460efef80051 not found: ID does not exist" Jul 07 17:52:46.848316 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.848275 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:46.850942 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:46.850921 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b0a4cbb-openshift-default-75994f6c8f-zfkph"] Jul 07 17:52:47.451395 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:47.451359 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" path="/var/lib/kubelet/pods/5b79ff4b-9414-42ab-b1b9-8a3070685d34/volumes" Jul 07 17:52:47.747195 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:47.747111 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:47.856405 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:47.856354 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:48.072873 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.072783 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:48.172698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.172659 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:48.249376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.249338 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:52:48.249887 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.249871 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" containerName="maas-api" Jul 07 17:52:48.249933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.249891 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" containerName="maas-api" Jul 07 17:52:48.249933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.249925 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" containerName="istio-proxy" Jul 07 17:52:48.249997 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.249934 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" containerName="istio-proxy" Jul 07 17:52:48.250032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.250016 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31348c6-c62b-4560-b3a8-6ce6ecec8278" containerName="maas-api" Jul 07 17:52:48.250032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.250027 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b79ff4b-9414-42ab-b1b9-8a3070685d34" containerName="istio-proxy" Jul 07 17:52:48.253247 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.253227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.256393 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.256369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9952c532-gw-openshift-default-dockercfg-jnrfv\"" Jul 07 17:52:48.265214 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.265184 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:52:48.298478 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.298433 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:48.336151 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336151 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336151 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336151 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtgc\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.336432 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.336364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.395671 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.395634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:48.437218 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtgc\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437218 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437335 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437720 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437848 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.437900 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.437878 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.438323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.438273 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.439823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.439791 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.440095 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.440075 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.446995 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.446974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtgc\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.447197 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.447177 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token\") pod \"e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.565669 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.565633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:48.737401 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.737373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:52:48.738259 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:52:48.738236 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1a24aa_ba95_47cb_b948_79dc3c525830.slice/crio-17db155b904166cebacf28f0ca3c2275f0c6b79c8cd61d91411441ebb65ba669 WatchSource:0}: Error finding container 17db155b904166cebacf28f0ca3c2275f0c6b79c8cd61d91411441ebb65ba669: Status 404 returned error can't find the container with id 17db155b904166cebacf28f0ca3c2275f0c6b79c8cd61d91411441ebb65ba669 Jul 07 17:52:48.740563 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.740523 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:48.740652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.740595 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:48.740652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.740624 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:52:48.838270 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.838230 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" event={"ID":"3e1a24aa-ba95-47cb-b948-79dc3c525830","Type":"ContainerStarted","Data":"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a"} Jul 07 17:52:48.838734 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.838326 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" event={"ID":"3e1a24aa-ba95-47cb-b948-79dc3c525830","Type":"ContainerStarted","Data":"17db155b904166cebacf28f0ca3c2275f0c6b79c8cd61d91411441ebb65ba669"} Jul 07 17:52:48.872266 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:48.872155 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" podStartSLOduration=0.872140567 podStartE2EDuration="872.140567ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:48.871225889 +0000 UTC m=+1833.991430221" watchObservedRunningTime="2026-07-07 17:52:48.872140567 +0000 UTC m=+1833.992344896" Jul 07 17:52:49.565641 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.565607 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:49.565805 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.565757 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:49.570853 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.570827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:49.685994 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.685959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:49.843111 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.843031 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:49.844361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.844340 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:52:49.956495 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:49.956459 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:50.323808 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:50.323779 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:50.427363 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:50.427326 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:51.603000 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:51.602969 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:52.099579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:52.099545 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:52:54.212466 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.212418 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:54.391097 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.391057 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:52:54.391386 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.391317 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" podUID="3e1a24aa-ba95-47cb-b948-79dc3c525830" containerName="istio-proxy" containerID="cri-o://be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a" gracePeriod=30 Jul 07 17:52:54.419328 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.419266 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:54.547683 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.547594 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:54.661270 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.661240 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:54.749581 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:54.749551 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:52:56.219503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:52:56.219461 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:00.437649 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.437622 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:53:00.560196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560110 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phtgc\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560152 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560176 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560200 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560347 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560392 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560426 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560500 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560489 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560521 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket\") pod \"3e1a24aa-ba95-47cb-b948-79dc3c525830\" (UID: \"3e1a24aa-ba95-47cb-b948-79dc3c525830\") " Jul 07 17:53:00.560781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560481 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:00.560781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560736 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:00.560879 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560790 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:53:00.561046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560976 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.561046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.560998 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.561046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.561011 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1a24aa-ba95-47cb-b948-79dc3c525830-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.561299 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.561140 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:00.561357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.561304 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:00.562890 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.562868 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:00.563124 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.563097 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:00.563124 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.563107 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc" (OuterVolumeSpecName: "kube-api-access-phtgc") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "kube-api-access-phtgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:00.563246 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.563171 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e1a24aa-ba95-47cb-b948-79dc3c525830" (UID: "3e1a24aa-ba95-47cb-b948-79dc3c525830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:53:00.661863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661821 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.661863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661856 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.661863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661868 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.662112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661881 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtgc\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-kube-api-access-phtgc\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.662112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661893 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.662112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.661906 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1a24aa-ba95-47cb-b948-79dc3c525830-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:00.792933 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.792895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:00.855767 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.855683 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:00.899343 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.899279 2569 generic.go:356] "Generic (PLEG): container finished" podID="3e1a24aa-ba95-47cb-b948-79dc3c525830" containerID="be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a" exitCode=0 Jul 07 17:53:00.899538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.899352 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" Jul 07 17:53:00.899538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.899374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" event={"ID":"3e1a24aa-ba95-47cb-b948-79dc3c525830","Type":"ContainerDied","Data":"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a"} Jul 07 17:53:00.899538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.899428 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht" event={"ID":"3e1a24aa-ba95-47cb-b948-79dc3c525830","Type":"ContainerDied","Data":"17db155b904166cebacf28f0ca3c2275f0c6b79c8cd61d91411441ebb65ba669"} Jul 07 17:53:00.899538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.899451 2569 scope.go:117] "RemoveContainer" containerID="be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a" Jul 07 17:53:00.909318 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.909275 2569 scope.go:117] "RemoveContainer" containerID="be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a" Jul 07 17:53:00.909610 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:53:00.909590 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a\": container with ID starting with be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a not found: ID does not exist" containerID="be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a" Jul 07 17:53:00.909674 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.909618 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a"} err="failed to get container status \"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a\": rpc error: code = NotFound desc = could not find container \"be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a\": container with ID starting with be61767da10c25582cc7e026f3acf050c0605ce47b4fce69847dc5ac14919b7a not found: ID does not exist" Jul 07 17:53:00.971041 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:00.970999 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:53:01.001104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.001074 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9952c532-gw-openshift-default-86b7d8f455-cfkht"] Jul 07 17:53:01.092552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.092509 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:01.092917 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.092897 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1a24aa-ba95-47cb-b948-79dc3c525830" containerName="istio-proxy" Jul 07 17:53:01.093000 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.092921 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1a24aa-ba95-47cb-b948-79dc3c525830" containerName="istio-proxy" Jul 07 17:53:01.093053 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.093046 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1a24aa-ba95-47cb-b948-79dc3c525830" containerName="istio-proxy" Jul 07 17:53:01.096389 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.096362 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.100331 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.100271 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f647f559-openshift-default-dockercfg-lm2t4\"" Jul 07 17:53:01.115566 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.115531 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:01.123727 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.123695 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:01.224707 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.224671 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:01.267652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjss\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.267850 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.268009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.268009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.267892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.322966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.322928 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:53:01.368996 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.368896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.368996 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.368931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.368996 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.368957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjss\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.368996 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.368996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369028 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369613 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369426 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369613 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369613 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369518 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.369934 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.369885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.370021 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.370002 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.371473 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.371450 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.371830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.371807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.378925 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.378891 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.379201 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.379175 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjss\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss\") pod \"e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.411093 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.411051 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:01.452061 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.452029 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1a24aa-ba95-47cb-b948-79dc3c525830" path="/var/lib/kubelet/pods/3e1a24aa-ba95-47cb-b948-79dc3c525830/volumes" Jul 07 17:53:01.559615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.559588 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:01.561977 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:53:01.561949 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56e7c9e_00d9_4df0_b4f2_c8b860666ad4.slice/crio-ce5ad2f7c03eb99f4c324d0c1255e40b6e2bedc3fbf21f90dca03a6271aa004c WatchSource:0}: Error finding container ce5ad2f7c03eb99f4c324d0c1255e40b6e2bedc3fbf21f90dca03a6271aa004c: Status 404 returned error can't find the container with id ce5ad2f7c03eb99f4c324d0c1255e40b6e2bedc3fbf21f90dca03a6271aa004c Jul 07 17:53:01.564300 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.564246 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:01.564400 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.564376 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:01.564448 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.564432 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:01.906964 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.906913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" event={"ID":"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4","Type":"ContainerStarted","Data":"91e64ac920a40f052fa5a78f49c50a16fb06701ae1e48378c05d6eb7ddfcfc81"} Jul 07 17:53:01.907186 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.906999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" event={"ID":"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4","Type":"ContainerStarted","Data":"ce5ad2f7c03eb99f4c324d0c1255e40b6e2bedc3fbf21f90dca03a6271aa004c"} Jul 07 17:53:01.930538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:01.930489 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" podStartSLOduration=0.930470342 podStartE2EDuration="930.470342ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:53:01.929083788 +0000 UTC m=+1847.049288139" watchObservedRunningTime="2026-07-07 17:53:01.930470342 +0000 UTC m=+1847.050674672" Jul 07 17:53:02.360530 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.360442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:02.412095 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.412056 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:02.417461 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.417429 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:02.595614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.595582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:02.644013 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.643974 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:02.646558 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.646535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.649482 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.649457 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f647f559-serving-cert\"" Jul 07 17:53:02.649943 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.649913 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:53:02.674306 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.672110 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:02.682386 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.682340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.682546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.682394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkm8b\" (UniqueName: \"kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.682546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.682517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.693239 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.693188 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:02.783394 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.783349 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.783612 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.783404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkm8b\" (UniqueName: \"kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.783612 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.783473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.783612 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:53:02.783519 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f647f559-serving-cert: secret "maas-api-e2e-ait-f647f559-serving-cert" not found Jul 07 17:53:02.783612 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:53:02.783603 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls podName:ac121301-cecd-4941-8dde-10d5b6a0a0b5 nodeName:}" failed. No retries permitted until 2026-07-07 17:53:03.283580931 +0000 UTC m=+1848.403785242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls") pod "maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" (UID: "ac121301-cecd-4941-8dde-10d5b6a0a0b5") : secret "maas-api-e2e-ait-f647f559-serving-cert" not found Jul 07 17:53:02.784361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.784335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.794782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.794753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkm8b\" (UniqueName: \"kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:02.909238 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.909143 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:02.912798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.912775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:02.913699 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:02.913679 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:03.289509 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.289395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:03.292024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.291999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") pod \"maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:03.424395 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.424365 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:03.524893 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.524859 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:03.558865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.558764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:03.704398 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.704372 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:03.706957 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:53:03.706932 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac121301_cecd_4941_8dde_10d5b6a0a0b5.slice/crio-5da7803e40f7795125ab59c70c883e7d4212f571557324325c6def9f59b11153 WatchSource:0}: Error finding container 5da7803e40f7795125ab59c70c883e7d4212f571557324325c6def9f59b11153: Status 404 returned error can't find the container with id 5da7803e40f7795125ab59c70c883e7d4212f571557324325c6def9f59b11153 Jul 07 17:53:03.918919 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:03.918881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" event={"ID":"ac121301-cecd-4941-8dde-10d5b6a0a0b5","Type":"ContainerStarted","Data":"5da7803e40f7795125ab59c70c883e7d4212f571557324325c6def9f59b11153"} Jul 07 17:53:04.643463 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:04.643432 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:04.925523 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:04.925491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" event={"ID":"ac121301-cecd-4941-8dde-10d5b6a0a0b5","Type":"ContainerStarted","Data":"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced"} Jul 07 17:53:04.926007 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:04.925548 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:04.944614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:04.944562 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" podStartSLOduration=2.519731449 podStartE2EDuration="2.944545122s" totalImagesPullingTime="424.813673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:53:04.943175467 +0000 UTC m=+1850.063379792" watchObservedRunningTime="2026-07-07 17:53:04.944545122 +0000 UTC m=+1850.064749469" Jul 07 17:53:07.678550 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:07.678497 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:08.084525 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.084442 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:08.084703 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.084671 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" podUID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" containerName="maas-api" containerID="cri-o://7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced" gracePeriod=30 Jul 07 17:53:08.114513 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.113986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:08.375668 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.375642 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:08.438748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.438711 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") pod \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " Jul 07 17:53:08.438748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.438752 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca\") pod \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " Jul 07 17:53:08.439025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.438841 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkm8b\" (UniqueName: \"kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b\") pod \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\" (UID: \"ac121301-cecd-4941-8dde-10d5b6a0a0b5\") " Jul 07 17:53:08.439183 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.439156 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac121301-cecd-4941-8dde-10d5b6a0a0b5" (UID: "ac121301-cecd-4941-8dde-10d5b6a0a0b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:53:08.440953 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.440929 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac121301-cecd-4941-8dde-10d5b6a0a0b5" (UID: "ac121301-cecd-4941-8dde-10d5b6a0a0b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:53:08.441264 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.441235 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b" (OuterVolumeSpecName: "kube-api-access-pkm8b") pod "ac121301-cecd-4941-8dde-10d5b6a0a0b5" (UID: "ac121301-cecd-4941-8dde-10d5b6a0a0b5"). InnerVolumeSpecName "kube-api-access-pkm8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:08.539690 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.539649 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac121301-cecd-4941-8dde-10d5b6a0a0b5-maas-api-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:08.539690 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.539681 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac121301-cecd-4941-8dde-10d5b6a0a0b5-openshift-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:08.539690 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.539694 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkm8b\" (UniqueName: \"kubernetes.io/projected/ac121301-cecd-4941-8dde-10d5b6a0a0b5-kube-api-access-pkm8b\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:08.947423 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.947384 2569 generic.go:356] "Generic (PLEG): container finished" podID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" containerID="7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced" exitCode=0 Jul 07 17:53:08.947903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.947431 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" Jul 07 17:53:08.947903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.947436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" event={"ID":"ac121301-cecd-4941-8dde-10d5b6a0a0b5","Type":"ContainerDied","Data":"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced"} Jul 07 17:53:08.947903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.947590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2" event={"ID":"ac121301-cecd-4941-8dde-10d5b6a0a0b5","Type":"ContainerDied","Data":"5da7803e40f7795125ab59c70c883e7d4212f571557324325c6def9f59b11153"} Jul 07 17:53:08.947903 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.947631 2569 scope.go:117] "RemoveContainer" containerID="7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced" Jul 07 17:53:08.957731 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.957699 2569 scope.go:117] "RemoveContainer" containerID="7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced" Jul 07 17:53:08.957965 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:53:08.957945 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced\": container with ID starting with 7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced not found: ID does not exist" containerID="7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced" Jul 07 17:53:08.958019 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.957973 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced"} err="failed to get container status \"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced\": rpc error: code = NotFound desc = could not find container \"7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced\": container with ID starting with 7dd3ed13e1f8a6309b986d01c0f24cfc49591a6d9765e662c5696cdfb9ef2ced not found: ID does not exist" Jul 07 17:53:08.970650 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.970626 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:08.975767 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:08.975744 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f647f559-6c45c67bf9-dw6d2"] Jul 07 17:53:09.451267 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:09.451233 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" path="/var/lib/kubelet/pods/ac121301-cecd-4941-8dde-10d5b6a0a0b5/volumes" Jul 07 17:53:13.914555 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:13.914518 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:13.914975 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:13.914735 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" podUID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" containerName="istio-proxy" containerID="cri-o://91e64ac920a40f052fa5a78f49c50a16fb06701ae1e48378c05d6eb7ddfcfc81" gracePeriod=30 Jul 07 17:53:13.983898 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:13.983865 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:14.140804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:14.140748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:14.979781 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:14.979740 2569 generic.go:356] "Generic (PLEG): container finished" podID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" containerID="91e64ac920a40f052fa5a78f49c50a16fb06701ae1e48378c05d6eb7ddfcfc81" exitCode=0 Jul 07 17:53:14.980169 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:14.979812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" event={"ID":"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4","Type":"ContainerDied","Data":"91e64ac920a40f052fa5a78f49c50a16fb06701ae1e48378c05d6eb7ddfcfc81"} Jul 07 17:53:15.083314 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.083273 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:15.196057 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196028 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196096 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196119 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnjss\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196140 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196171 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196223 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196193 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196331 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket\") pod \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\" (UID: \"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4\") " Jul 07 17:53:15.196499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196339 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:15.196705 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196632 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.196779 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196748 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:53:15.196856 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196829 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:15.196856 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.196847 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:15.197176 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.197148 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:15.198599 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.198573 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:15.198702 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.198679 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:53:15.198862 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.198841 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:15.198953 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.198865 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss" (OuterVolumeSpecName: "kube-api-access-vnjss") pod "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" (UID: "c56e7c9e-00d9-4df0-b4f2-c8b860666ad4"). InnerVolumeSpecName "kube-api-access-vnjss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:15.297196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297155 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjss\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-kube-api-access-vnjss\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297190 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297199 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297196 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297207 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297215 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297224 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297232 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.297497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.297240 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:15.985442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.985410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" event={"ID":"c56e7c9e-00d9-4df0-b4f2-c8b860666ad4","Type":"ContainerDied","Data":"ce5ad2f7c03eb99f4c324d0c1255e40b6e2bedc3fbf21f90dca03a6271aa004c"} Jul 07 17:53:15.985442 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.985442 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn" Jul 07 17:53:15.985909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:15.985471 2569 scope.go:117] "RemoveContainer" containerID="91e64ac920a40f052fa5a78f49c50a16fb06701ae1e48378c05d6eb7ddfcfc81" Jul 07 17:53:16.007054 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:16.007021 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:16.011043 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:16.011016 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f647f559-openshift-default-684d8fdbc5-nkfcn"] Jul 07 17:53:17.458178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:17.458144 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" path="/var/lib/kubelet/pods/c56e7c9e-00d9-4df0-b4f2-c8b860666ad4/volumes" Jul 07 17:53:20.282497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.282443 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:20.384972 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.384937 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:20.420865 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.420831 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:20.421435 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421414 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" containerName="maas-api" Jul 07 17:53:20.421491 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421439 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" containerName="maas-api" Jul 07 17:53:20.421491 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421464 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" containerName="istio-proxy" Jul 07 17:53:20.421491 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421472 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" containerName="istio-proxy" Jul 07 17:53:20.421588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421564 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac121301-cecd-4941-8dde-10d5b6a0a0b5" containerName="maas-api" Jul 07 17:53:20.421588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.421577 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56e7c9e-00d9-4df0-b4f2-c8b860666ad4" containerName="istio-proxy" Jul 07 17:53:20.425047 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.425027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.427943 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.427922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f27f5913-openshift-default-dockercfg-s9smh\"" Jul 07 17:53:20.435609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.435582 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:20.470921 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.470889 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:20.546661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gfb\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.546661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.546867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546681 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.546867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.546867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.546867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.547083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546905 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.547083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.546998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.547083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.547046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648073 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648073 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648336 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gfb\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648579 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648783 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648783 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.648783 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.648777 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.649050 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.649026 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.650846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.650822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.650990 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.650970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.657512 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.657485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.657827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.657810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gfb\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb\") pod \"e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.738514 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.738481 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:20.892649 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.892620 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:20.894017 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:53:20.893987 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70744be9_44f9_490e_ab17_7693cec964f3.slice/crio-badc68a774c294b73cadde430cdf32cc213a4ba10cc3b9407f31de77b1b07f1a WatchSource:0}: Error finding container badc68a774c294b73cadde430cdf32cc213a4ba10cc3b9407f31de77b1b07f1a: Status 404 returned error can't find the container with id badc68a774c294b73cadde430cdf32cc213a4ba10cc3b9407f31de77b1b07f1a Jul 07 17:53:20.896418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.896383 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:20.896497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.896453 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:20.896497 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:20.896483 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:53:21.014635 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.014589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" event={"ID":"70744be9-44f9-490e-ab17-7693cec964f3","Type":"ContainerStarted","Data":"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613"} Jul 07 17:53:21.014811 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.014658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" event={"ID":"70744be9-44f9-490e-ab17-7693cec964f3","Type":"ContainerStarted","Data":"badc68a774c294b73cadde430cdf32cc213a4ba10cc3b9407f31de77b1b07f1a"} Jul 07 17:53:21.039103 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.039043 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" podStartSLOduration=1.039018792 podStartE2EDuration="1.039018792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:53:21.035895454 +0000 UTC m=+1866.156099783" watchObservedRunningTime="2026-07-07 17:53:21.039018792 +0000 UTC m=+1866.159223124" Jul 07 17:53:21.592361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.592300 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:21.687575 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.687530 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:21.739059 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.739029 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:21.744414 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.744388 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:21.760475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.760435 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:21.860946 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:21.860848 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:22.020277 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:22.020242 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:22.021182 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:22.021156 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:22.265194 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:22.265154 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:22.368229 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:22.368146 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:24.387795 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:24.387759 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:53:24.640763 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:24.640650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:26.059107 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.059067 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:26.209800 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.209753 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:26.210017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.209983 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" podUID="70744be9-44f9-490e-ab17-7693cec964f3" containerName="istio-proxy" containerID="cri-o://cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613" gracePeriod=30 Jul 07 17:53:26.242156 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.242122 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:26.342440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.342361 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:26.491081 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:26.491047 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:28.058536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:28.058503 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:32.578050 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.577991 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:32.668361 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668318 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668382 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668418 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668505 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668536 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668531 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668564 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2gfb\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668592 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668635 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token\") pod \"70744be9-44f9-490e-ab17-7693cec964f3\" (UID: \"70744be9-44f9-490e-ab17-7693cec964f3\") " Jul 07 17:53:32.668796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668691 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:32.668796 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668771 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:32.669105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668936 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.669105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.668954 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.669105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.669018 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:53:32.669243 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.669134 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:32.669839 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.669808 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:32.670945 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.670923 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:53:32.671067 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.670953 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb" (OuterVolumeSpecName: "kube-api-access-t2gfb") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "kube-api-access-t2gfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:32.671333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.671310 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:53:32.671440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.671355 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "70744be9-44f9-490e-ab17-7693cec964f3" (UID: "70744be9-44f9-490e-ab17-7693cec964f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:32.769614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769578 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769609 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769614 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769619 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70744be9-44f9-490e-ab17-7693cec964f3-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769629 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gfb\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-kube-api-access-t2gfb\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769638 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70744be9-44f9-490e-ab17-7693cec964f3-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769646 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70744be9-44f9-490e-ab17-7693cec964f3-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.769858 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.769653 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70744be9-44f9-490e-ab17-7693cec964f3-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:53:32.996791 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:32.996754 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:33.077732 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.077698 2569 generic.go:356] "Generic (PLEG): container finished" podID="70744be9-44f9-490e-ab17-7693cec964f3" containerID="cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613" exitCode=0 Jul 07 17:53:33.077912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.077745 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" Jul 07 17:53:33.077912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.077786 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" event={"ID":"70744be9-44f9-490e-ab17-7693cec964f3","Type":"ContainerDied","Data":"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613"} Jul 07 17:53:33.077912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.077841 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b" event={"ID":"70744be9-44f9-490e-ab17-7693cec964f3","Type":"ContainerDied","Data":"badc68a774c294b73cadde430cdf32cc213a4ba10cc3b9407f31de77b1b07f1a"} Jul 07 17:53:33.077912 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.077873 2569 scope.go:117] "RemoveContainer" containerID="cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613" Jul 07 17:53:33.087341 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.087323 2569 scope.go:117] "RemoveContainer" containerID="cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613" Jul 07 17:53:33.087629 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:53:33.087609 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613\": container with ID starting with cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613 not found: ID does not exist" containerID="cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613" Jul 07 17:53:33.087679 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.087639 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613"} err="failed to get container status \"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613\": rpc error: code = NotFound desc = could not find container \"cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613\": container with ID starting with cc46ee026527c7ff5b35e293f6bc0882d5ee0ff3eb2d3794627849f63e484613 not found: ID does not exist" Jul 07 17:53:33.100661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.100628 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:53:33.112656 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.112628 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:33.125415 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.125381 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f27f5913-openshift-default-68f448d89f-fnc7b"] Jul 07 17:53:33.131727 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.131700 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:33.451120 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.451083 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70744be9-44f9-490e-ab17-7693cec964f3" path="/var/lib/kubelet/pods/70744be9-44f9-490e-ab17-7693cec964f3/volumes" Jul 07 17:53:33.509357 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.509322 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:33.643693 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.643659 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:33.746782 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:33.746692 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:35.248643 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:35.248605 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:41.863971 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:41.863889 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:41.932623 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:41.932593 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:42.054603 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:42.054574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:42.158323 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:42.158277 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:53:42.595696 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:53:42.595610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:08.667011 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:08.666964 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:14.998753 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:14.998719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:15.064116 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:15.064082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:15.157887 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:15.157849 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:15.256745 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:15.256655 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:25.206466 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:25.206429 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:31.221543 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:31.221501 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:37.294030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:37.293990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:37.370562 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:37.370528 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:37.461195 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:37.461163 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:37.561721 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:37.561627 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:48.090107 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:48.090071 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:49.040669 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:49.040630 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:55.516897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:55.516866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:55.631142 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:55.631108 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:55.727221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:55.727187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:55.831995 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:55.831876 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.850731 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:56.850693 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.946506 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:56.946466 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:57.056949 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:57.056914 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:58.095724 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:58.095688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:54:58.906593 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:54:58.906557 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:01.477813 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:01.477778 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:05.544953 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:05.544896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:05.619697 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:05.619659 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:55:05.763804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:05.763768 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:06.318107 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:06.318020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.175066 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.175024 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.275784 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.275748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.472299 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.472198 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.589744 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.589712 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:55:13.804751 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.804669 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:13.805226 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.805212 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70744be9-44f9-490e-ab17-7693cec964f3" containerName="istio-proxy" Jul 07 17:55:13.805267 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.805229 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="70744be9-44f9-490e-ab17-7693cec964f3" containerName="istio-proxy" Jul 07 17:55:13.805348 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.805337 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="70744be9-44f9-490e-ab17-7693cec964f3" containerName="istio-proxy" Jul 07 17:55:13.808703 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.808684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.828334 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.828312 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12123822-openshift-default-dockercfg-cgq94\"" Jul 07 17:55:13.864549 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.864518 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.955183 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955146 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:13.955564 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955618 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955618 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrh4\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955699 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955733 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955769 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955818 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955894 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:13.955929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:13.955904 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056587 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056615 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrh4\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.056931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.056800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.057193 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.057113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.057193 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.057163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.057398 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.057375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.057482 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.057405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.057703 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.057539 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.059324 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.059300 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.059435 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.059375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.073136 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.073111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.073253 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.073231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrh4\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4\") pod \"e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.119872 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.119840 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:14.290827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.290792 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:14.294491 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:55:14.294465 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2db815b_5cf7_4b27_a858_980397f0295b.slice/crio-1741c53d9fda33567c5ad3c96dbf714dd508c1502e1f40ecc0221e5c1ffbb15d WatchSource:0}: Error finding container 1741c53d9fda33567c5ad3c96dbf714dd508c1502e1f40ecc0221e5c1ffbb15d: Status 404 returned error can't find the container with id 1741c53d9fda33567c5ad3c96dbf714dd508c1502e1f40ecc0221e5c1ffbb15d Jul 07 17:55:14.297015 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.296984 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:14.297085 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.297050 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:14.297121 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.297083 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:14.562743 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.562646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerStarted","Data":"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b"} Jul 07 17:55:14.562743 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.562718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerStarted","Data":"1741c53d9fda33567c5ad3c96dbf714dd508c1502e1f40ecc0221e5c1ffbb15d"} Jul 07 17:55:14.591596 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.591528 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podStartSLOduration=1.591505035 podStartE2EDuration="1.591505035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:14.589777735 +0000 UTC m=+1979.709982075" watchObservedRunningTime="2026-07-07 17:55:14.591505035 +0000 UTC m=+1979.711709363" Jul 07 17:55:14.995852 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:14.995823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:15.120553 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:15.120520 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:15.125629 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:15.125602 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:15.567736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:15.567702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:15.568574 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:15.568555 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:17.510259 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.510224 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.607907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.607873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.609174 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.609145 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:17.613044 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.613024 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.615844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.615823 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-12123822-serving-cert\"" Jul 07 17:55:17.615962 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.615937 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:55:17.623504 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.623480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:17.685798 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.685761 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.790011 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.789895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.793134 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.793108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrlq7\" (UniqueName: \"kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.793221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.793173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.793221 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.793200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.894155 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.894121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.894376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.894162 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.894376 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.894206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrlq7\" (UniqueName: \"kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.894855 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.894829 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.896827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.896806 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.904537 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.904515 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrlq7\" (UniqueName: \"kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7\") pod \"maas-api-e2e-mt-12123822-85b5b95847-dh7h9\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:17.925527 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:17.925490 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:18.076907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:18.076880 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:18.078856 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:55:18.078830 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4105819c_708b_4a83_9843_256a93d7ecd5.slice/crio-df039012e87d9c3f2a0e3abf93ed098de7a0cc73d3e34b49536f96cd013d89e2 WatchSource:0}: Error finding container df039012e87d9c3f2a0e3abf93ed098de7a0cc73d3e34b49536f96cd013d89e2: Status 404 returned error can't find the container with id df039012e87d9c3f2a0e3abf93ed098de7a0cc73d3e34b49536f96cd013d89e2 Jul 07 17:55:18.249761 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:18.249729 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:18.346095 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:18.346007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:18.582219 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:18.582189 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" event={"ID":"4105819c-708b-4a83-9843-256a93d7ecd5","Type":"ContainerStarted","Data":"df039012e87d9c3f2a0e3abf93ed098de7a0cc73d3e34b49536f96cd013d89e2"} Jul 07 17:55:19.589044 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:19.589007 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" event={"ID":"4105819c-708b-4a83-9843-256a93d7ecd5","Type":"ContainerStarted","Data":"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94"} Jul 07 17:55:19.589700 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:19.589149 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:19.614447 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:19.614400 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" podStartSLOduration=2.150052078 podStartE2EDuration="2.61438647s" totalImagesPullingTime="464.334392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:19.613710031 +0000 UTC m=+1984.733914364" watchObservedRunningTime="2026-07-07 17:55:19.61438647 +0000 UTC m=+1984.734590825" Jul 07 17:55:21.788276 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:21.788238 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:55:22.042960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:22.042853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:22.773119 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:22.773078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:22.883629 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:22.883596 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:23.273342 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:23.273299 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:25.604689 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:25.604642 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:30.374661 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.374626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:30.472083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.472050 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:30.568859 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.568824 2569 patch_prober.go:28] interesting pod/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 07 17:55:30.569023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.568887 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 07 17:55:30.653736 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.653703 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2db815b-5cf7-4b27-a858-980397f0295b" containerID="cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b" exitCode=0 Jul 07 17:55:30.653908 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.653763 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerDied","Data":"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b"} Jul 07 17:55:30.654147 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.654130 2569 scope.go:117] "RemoveContainer" containerID="cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b" Jul 07 17:55:30.655331 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.655297 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:30.655409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.655398 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:30.655451 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:30.655437 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:55:31.016538 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.016427 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.458205 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.458173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.611321 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.611255 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:31.612399 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.612347 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" podUID="4105819c-708b-4a83-9843-256a93d7ecd5" containerName="maas-api" containerID="cri-o://35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94" gracePeriod=30 Jul 07 17:55:31.613135 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.612986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.660649 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.660619 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerStarted","Data":"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e"} Jul 07 17:55:31.879202 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.879178 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:31.920737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.920697 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca\") pod \"4105819c-708b-4a83-9843-256a93d7ecd5\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " Jul 07 17:55:31.920941 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.920755 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrlq7\" (UniqueName: \"kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7\") pod \"4105819c-708b-4a83-9843-256a93d7ecd5\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " Jul 07 17:55:31.920941 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.920812 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls\") pod \"4105819c-708b-4a83-9843-256a93d7ecd5\" (UID: \"4105819c-708b-4a83-9843-256a93d7ecd5\") " Jul 07 17:55:31.922016 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.921913 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4105819c-708b-4a83-9843-256a93d7ecd5" (UID: "4105819c-708b-4a83-9843-256a93d7ecd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:55:31.924526 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.924494 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7" (OuterVolumeSpecName: "kube-api-access-zrlq7") pod "4105819c-708b-4a83-9843-256a93d7ecd5" (UID: "4105819c-708b-4a83-9843-256a93d7ecd5"). InnerVolumeSpecName "kube-api-access-zrlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:31.924715 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:31.924689 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4105819c-708b-4a83-9843-256a93d7ecd5" (UID: "4105819c-708b-4a83-9843-256a93d7ecd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:55:32.022210 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.022115 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105819c-708b-4a83-9843-256a93d7ecd5-openshift-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:32.022210 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.022159 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrlq7\" (UniqueName: \"kubernetes.io/projected/4105819c-708b-4a83-9843-256a93d7ecd5-kube-api-access-zrlq7\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:32.022210 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.022175 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4105819c-708b-4a83-9843-256a93d7ecd5-maas-api-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:32.121139 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.121094 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:32.121424 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.121399 2569 patch_prober.go:28] interesting pod/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 07 17:55:32.121479 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.121452 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 07 17:55:32.667015 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.666975 2569 generic.go:356] "Generic (PLEG): container finished" podID="4105819c-708b-4a83-9843-256a93d7ecd5" containerID="35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94" exitCode=0 Jul 07 17:55:32.667586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.667023 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" Jul 07 17:55:32.667586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.667057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" event={"ID":"4105819c-708b-4a83-9843-256a93d7ecd5","Type":"ContainerDied","Data":"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94"} Jul 07 17:55:32.667586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.667109 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9" event={"ID":"4105819c-708b-4a83-9843-256a93d7ecd5","Type":"ContainerDied","Data":"df039012e87d9c3f2a0e3abf93ed098de7a0cc73d3e34b49536f96cd013d89e2"} Jul 07 17:55:32.667586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.667132 2569 scope.go:117] "RemoveContainer" containerID="35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94" Jul 07 17:55:32.676978 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.676960 2569 scope.go:117] "RemoveContainer" containerID="35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94" Jul 07 17:55:32.677252 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:55:32.677235 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94\": container with ID starting with 35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94 not found: ID does not exist" containerID="35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94" Jul 07 17:55:32.677317 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.677260 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94"} err="failed to get container status \"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94\": rpc error: code = NotFound desc = could not find container \"35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94\": container with ID starting with 35d9e86f41df2c0a06d6e8c507427b1ae2a8995301300deeaca0cdc15fafaf94 not found: ID does not exist" Jul 07 17:55:32.713387 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.713199 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:32.717017 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:32.716900 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-12123822-85b5b95847-dh7h9"] Jul 07 17:55:33.120746 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:33.120654 2569 patch_prober.go:28] interesting pod/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 07 17:55:33.120746 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:33.120715 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 07 17:55:33.451656 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:33.451620 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4105819c-708b-4a83-9843-256a93d7ecd5" path="/var/lib/kubelet/pods/4105819c-708b-4a83-9843-256a93d7ecd5/volumes" Jul 07 17:55:34.120590 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:34.120546 2569 patch_prober.go:28] interesting pod/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 07 17:55:34.120995 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:34.120625 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 07 17:55:35.120390 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:35.120348 2569 patch_prober.go:28] interesting pod/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 07 17:55:35.120572 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:35.120423 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 07 17:55:35.482194 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:35.482157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:55:36.124586 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:36.124557 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:36.124910 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:36.124888 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:36.125507 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:36.125487 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:37.682025 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:37.681990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:37.754712 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:37.754672 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:37.954783 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:37.954697 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:37.968950 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:37.968923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:38.258956 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:38.258866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:38.385029 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:38.384995 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:38.486112 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:38.486079 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:38.682835 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:38.682804 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:38.696823 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:38.696794 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" containerID="cri-o://032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e" gracePeriod=30 Jul 07 17:55:41.770261 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:41.770225 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:41.871979 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:41.871950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:43.953016 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:43.952985 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:44.038104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038010 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038060 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038079 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038104 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038098 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038278 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnrh4\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038369 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038396 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038431 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:44.038475 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038460 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs\") pod \"b2db815b-5cf7-4b27-a858-980397f0295b\" (UID: \"b2db815b-5cf7-4b27-a858-980397f0295b\") " Jul 07 17:55:44.038804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038523 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:55:44.038804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038659 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:44.038804 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038781 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.039132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038853 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:44.039132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038975 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.039132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.038998 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2db815b-5cf7-4b27-a858-980397f0295b-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.039319 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.039278 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:44.040454 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.040425 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:55:44.040560 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.040531 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:44.041032 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.041014 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:44.041092 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.041035 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4" (OuterVolumeSpecName: "kube-api-access-dnrh4") pod "b2db815b-5cf7-4b27-a858-980397f0295b" (UID: "b2db815b-5cf7-4b27-a858-980397f0295b"). InnerVolumeSpecName "kube-api-access-dnrh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:44.140240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140199 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.140240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140233 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2db815b-5cf7-4b27-a858-980397f0295b-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.140240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140247 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrh4\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-kube-api-access-dnrh4\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.140499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140260 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.140499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140272 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2db815b-5cf7-4b27-a858-980397f0295b-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.140499 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.140301 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2db815b-5cf7-4b27-a858-980397f0295b-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:55:44.725802 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.725767 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2db815b-5cf7-4b27-a858-980397f0295b" containerID="032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e" exitCode=0 Jul 07 17:55:44.726006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.725813 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerDied","Data":"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e"} Jul 07 17:55:44.726006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.725819 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" Jul 07 17:55:44.726006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.725854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl" event={"ID":"b2db815b-5cf7-4b27-a858-980397f0295b","Type":"ContainerDied","Data":"1741c53d9fda33567c5ad3c96dbf714dd508c1502e1f40ecc0221e5c1ffbb15d"} Jul 07 17:55:44.726006 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.725876 2569 scope.go:117] "RemoveContainer" containerID="032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e" Jul 07 17:55:44.735042 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.735023 2569 scope.go:117] "RemoveContainer" containerID="cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b" Jul 07 17:55:44.744951 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.744929 2569 scope.go:117] "RemoveContainer" containerID="032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e" Jul 07 17:55:44.745190 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:55:44.745171 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e\": container with ID starting with 032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e not found: ID does not exist" containerID="032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e" Jul 07 17:55:44.745264 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.745198 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e"} err="failed to get container status \"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e\": rpc error: code = NotFound desc = could not find container \"032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e\": container with ID starting with 032a1aae7d436ce1eb36d2eb3861838824fd5539fb4bccdd2974c32bcc9a071e not found: ID does not exist" Jul 07 17:55:44.745264 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.745216 2569 scope.go:117] "RemoveContainer" containerID="cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b" Jul 07 17:55:44.745493 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:55:44.745474 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b\": container with ID starting with cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b not found: ID does not exist" containerID="cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b" Jul 07 17:55:44.745535 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.745499 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b"} err="failed to get container status \"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b\": rpc error: code = NotFound desc = could not find container \"cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b\": container with ID starting with cbebe6b2e545558225bb82910fac149fc83ee6d1afefe1408bf814301830b26b not found: ID does not exist" Jul 07 17:55:44.774803 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.774780 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:44.780276 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:44.780256 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12123822-openshift-default-554f5bc9ff-gnxkl"] Jul 07 17:55:45.457011 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:45.456974 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" path="/var/lib/kubelet/pods/b2db815b-5cf7-4b27-a858-980397f0295b/volumes" Jul 07 17:55:45.538590 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:45.538559 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:45.643446 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:45.643412 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:45.747968 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:45.747889 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:45.974371 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:45.974338 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:52.275074 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:52.275038 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:52.343863 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:52.343831 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:59.437640 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.437608 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:59.807378 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807253 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:55:59.807637 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807623 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:55:59.807685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807639 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:55:59.807685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807658 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4105819c-708b-4a83-9843-256a93d7ecd5" containerName="maas-api" Jul 07 17:55:59.807685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807663 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4105819c-708b-4a83-9843-256a93d7ecd5" containerName="maas-api" Jul 07 17:55:59.807685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807671 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:55:59.807685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807676 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:55:59.807840 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807733 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:55:59.807840 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.807742 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4105819c-708b-4a83-9843-256a93d7ecd5" containerName="maas-api" Jul 07 17:55:59.810844 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.810827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.814144 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.814123 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb345660-openshift-default-dockercfg-g2xn9\"" Jul 07 17:55:59.824125 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.824098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:59.826261 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.826237 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:55:59.888771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888890 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.888988 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.888975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.889177 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.889007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxgv\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.987851 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.987814 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:59.989702 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.989779 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.989779 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.989859 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxgv\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.989907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.989907 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.989929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990132 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990224 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990345 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990320 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990540 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990516 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990630 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990630 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990578 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.990630 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.990600 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.992839 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.992814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.993035 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.993019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:55:59.999719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:55:59.999696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxgv\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:00.000827 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.000804 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token\") pod \"e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:00.121422 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.121320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:00.277882 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.277844 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:56:00.281967 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:56:00.281938 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0adb901_04ba_48f0_bd2e_f2d4a980b45a.slice/crio-587f68060ec2b355c197f9cbc1bae194bbe445ea7fac31d79adc42a67e53a0ef WatchSource:0}: Error finding container 587f68060ec2b355c197f9cbc1bae194bbe445ea7fac31d79adc42a67e53a0ef: Status 404 returned error can't find the container with id 587f68060ec2b355c197f9cbc1bae194bbe445ea7fac31d79adc42a67e53a0ef Jul 07 17:56:00.284078 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.284046 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:00.284168 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.284112 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:00.284168 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.284143 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:00.803402 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.803358 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerStarted","Data":"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef"} Jul 07 17:56:00.803813 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.803423 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerStarted","Data":"587f68060ec2b355c197f9cbc1bae194bbe445ea7fac31d79adc42a67e53a0ef"} Jul 07 17:56:00.833462 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:00.833413 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podStartSLOduration=1.8333980300000001 podStartE2EDuration="1.83339803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:56:00.832085544 +0000 UTC m=+2025.952289874" watchObservedRunningTime="2026-07-07 17:56:00.83339803 +0000 UTC m=+2025.953602359" Jul 07 17:56:01.122145 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:01.122047 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:01.127077 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:01.127052 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:01.808246 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:01.808218 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:01.809345 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:01.809318 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:02.943013 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:02.942978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.042590 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.042557 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.200613 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.200531 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:03.200962 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.200949 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2db815b-5cf7-4b27-a858-980397f0295b" containerName="istio-proxy" Jul 07 17:56:03.204119 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.204103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.208194 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.208172 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-cb345660-serving-cert\"" Jul 07 17:56:03.208332 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.208274 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:56:03.219957 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.219932 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:03.235275 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.235245 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.322952 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.322914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgq25\" (UniqueName: \"kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.323140 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.322962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.323140 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.323042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.341024 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.340988 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.423868 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.423837 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.424046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.423895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgq25\" (UniqueName: \"kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.424046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.423934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.424717 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.424630 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.426509 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.426486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.434421 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.434395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgq25\" (UniqueName: \"kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25\") pod \"maas-api-e2e-mt-cb345660-866f796fff-wz62r\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.516510 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.516434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:03.661098 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.661070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:03.663191 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:56:03.663163 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6647cb3f_8648_4c60_a0d6_1bdfb199de57.slice/crio-c8450ab6fbee8760a9d745b66fb6df1ff954c19d1f2224465a618859e176c8dc WatchSource:0}: Error finding container c8450ab6fbee8760a9d745b66fb6df1ff954c19d1f2224465a618859e176c8dc: Status 404 returned error can't find the container with id c8450ab6fbee8760a9d745b66fb6df1ff954c19d1f2224465a618859e176c8dc Jul 07 17:56:03.819830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.819738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" event={"ID":"6647cb3f-8648-4c60-a0d6-1bdfb199de57","Type":"ContainerStarted","Data":"c8450ab6fbee8760a9d745b66fb6df1ff954c19d1f2224465a618859e176c8dc"} Jul 07 17:56:03.934758 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:03.934731 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:04.035093 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:04.035053 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:04.827029 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:04.826989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" event={"ID":"6647cb3f-8648-4c60-a0d6-1bdfb199de57","Type":"ContainerStarted","Data":"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b"} Jul 07 17:56:04.827191 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:04.827149 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:07.399030 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:07.398993 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:08.585715 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:08.585661 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" podStartSLOduration=5.112328011 podStartE2EDuration="5.585644096s" totalImagesPullingTime="473.316085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:56:04.850379089 +0000 UTC m=+2029.970583430" watchObservedRunningTime="2026-07-07 17:56:08.585644096 +0000 UTC m=+2033.705848434" Jul 07 17:56:08.586888 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:08.586870 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:56:10.838545 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:10.838511 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:15.238228 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:15.238187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:16.809841 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.809808 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 17:56:16.810265 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.809874 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 17:56:16.894985 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.894956 2569 generic.go:356] "Generic (PLEG): container finished" podID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerID="85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef" exitCode=0 Jul 07 17:56:16.895150 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.895001 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerDied","Data":"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef"} Jul 07 17:56:16.895428 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.895413 2569 scope.go:117] "RemoveContainer" containerID="85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef" Jul 07 17:56:16.896382 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.896353 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:16.896444 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.896420 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:16.896479 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:16.896449 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:56:17.905331 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:17.905261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerStarted","Data":"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df"} Jul 07 17:56:18.121991 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:18.121946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:18.122270 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:18.122243 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 17:56:18.122392 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:18.122324 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 17:56:19.122501 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:19.122459 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 17:56:19.122897 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:19.122515 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 17:56:20.122526 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:20.122483 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 17:56:20.122917 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:20.122552 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 17:56:20.500085 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:20.500040 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:56:21.122581 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:21.122543 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 17:56:21.122974 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:21.122629 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 17:56:22.125942 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:22.125911 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:22.126352 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:22.126176 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:22.126829 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:22.126811 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:56:34.781686 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:34.781649 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:56:40.152688 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.152610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:40.737625 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.737586 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:40.737821 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.737781 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerName="maas-api" containerID="cri-o://31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b" gracePeriod=30 Jul 07 17:56:40.743665 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.743640 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:40.833492 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.833452 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.47:8443/health\": dial tcp 10.132.0.47:8443: connect: connection refused" Jul 07 17:56:40.999787 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:40.999713 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:41.023275 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.023242 2569 generic.go:356] "Generic (PLEG): container finished" podID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerID="31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b" exitCode=0 Jul 07 17:56:41.023471 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.023304 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" Jul 07 17:56:41.023471 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.023325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" event={"ID":"6647cb3f-8648-4c60-a0d6-1bdfb199de57","Type":"ContainerDied","Data":"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b"} Jul 07 17:56:41.023471 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.023394 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r" event={"ID":"6647cb3f-8648-4c60-a0d6-1bdfb199de57","Type":"ContainerDied","Data":"c8450ab6fbee8760a9d745b66fb6df1ff954c19d1f2224465a618859e176c8dc"} Jul 07 17:56:41.023471 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.023423 2569 scope.go:117] "RemoveContainer" containerID="31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b" Jul 07 17:56:41.033140 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.033118 2569 scope.go:117] "RemoveContainer" containerID="31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b" Jul 07 17:56:41.033484 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:56:41.033460 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b\": container with ID starting with 31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b not found: ID does not exist" containerID="31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b" Jul 07 17:56:41.033558 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.033497 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b"} err="failed to get container status \"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b\": rpc error: code = NotFound desc = could not find container \"31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b\": container with ID starting with 31c56b059e9c9ac62d72e32257b318f903ccd16d167c2d4c3ad00957366eb58b not found: ID does not exist" Jul 07 17:56:41.090476 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.090439 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca\") pod \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " Jul 07 17:56:41.090476 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.090483 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgq25\" (UniqueName: \"kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25\") pod \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " Jul 07 17:56:41.090703 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.090557 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls\") pod \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\" (UID: \"6647cb3f-8648-4c60-a0d6-1bdfb199de57\") " Jul 07 17:56:41.090819 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.090794 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6647cb3f-8648-4c60-a0d6-1bdfb199de57" (UID: "6647cb3f-8648-4c60-a0d6-1bdfb199de57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:56:41.092748 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.092713 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6647cb3f-8648-4c60-a0d6-1bdfb199de57" (UID: "6647cb3f-8648-4c60-a0d6-1bdfb199de57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:56:41.092867 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.092785 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25" (OuterVolumeSpecName: "kube-api-access-xgq25") pod "6647cb3f-8648-4c60-a0d6-1bdfb199de57" (UID: "6647cb3f-8648-4c60-a0d6-1bdfb199de57"). InnerVolumeSpecName "kube-api-access-xgq25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:56:41.191567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.191522 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6647cb3f-8648-4c60-a0d6-1bdfb199de57-maas-api-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:56:41.191567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.191562 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6647cb3f-8648-4c60-a0d6-1bdfb199de57-openshift-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:56:41.191567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.191576 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgq25\" (UniqueName: \"kubernetes.io/projected/6647cb3f-8648-4c60-a0d6-1bdfb199de57-kube-api-access-xgq25\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:56:41.351435 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.351406 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:41.355747 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.355721 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-cb345660-866f796fff-wz62r"] Jul 07 17:56:41.451046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:41.451009 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" path="/var/lib/kubelet/pods/6647cb3f-8648-4c60-a0d6-1bdfb199de57/volumes" Jul 07 17:56:44.725452 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:44.725416 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:56:46.405753 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:46.405720 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:51.501641 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:51.501600 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:55.284710 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:56:55.284675 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:57:15.549404 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:15.549377 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:57:15.555749 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:15.555725 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 17:57:27.989577 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:27.989545 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:29.038652 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:29.038618 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:31.194178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:31.194145 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:57:35.260928 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:35.260895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:35.329236 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:35.329201 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:35.688832 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:35.688801 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:57:35.689042 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:35.689018 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" containerID="cri-o://159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df" gracePeriod=30 Jul 07 17:57:35.736530 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:35.736499 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:36.016001 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:36.015925 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:37.128612 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:37.128567 2569 patch_prober.go:28] interesting pod/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 17:57:37.129009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:37.128644 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:57:37.260765 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:37.260728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:37.356609 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:37.356575 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:37.455540 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:37.455513 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:38.427712 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:38.427681 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:38.532892 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:38.532857 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:38.628814 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:38.628784 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:40.217046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:40.217012 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:40.947764 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:40.947743 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:57:41.108943 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.108852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:57:41.140533 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140503 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140687 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140687 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140638 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140687 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krxgv\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140699 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140723 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140750 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140793 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.140854 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140837 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy\") pod \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\" (UID: \"c0adb901-04ba-48f0-bd2e-f2d4a980b45a\") " Jul 07 17:57:41.141092 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.140965 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:41.141153 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141104 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:41.141212 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141145 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:41.141212 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141183 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.141212 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141200 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.141418 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141340 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:41.141810 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.141787 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:41.143046 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.143018 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:41.143145 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.143061 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv" (OuterVolumeSpecName: "kube-api-access-krxgv") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "kube-api-access-krxgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:41.143574 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.143553 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:41.143681 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.143572 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0adb901-04ba-48f0-bd2e-f2d4a980b45a" (UID: "c0adb901-04ba-48f0-bd2e-f2d4a980b45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:57:41.241875 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.241836 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.241967 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxgv\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-kube-api-access-krxgv\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.242002 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.242020 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.242035 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.242049 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.242312 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.242062 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0adb901-04ba-48f0-bd2e-f2d4a980b45a-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 17:57:41.312525 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.312491 2569 generic.go:356] "Generic (PLEG): container finished" podID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerID="159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df" exitCode=0 Jul 07 17:57:41.312704 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.312549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerDied","Data":"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df"} Jul 07 17:57:41.312704 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.312597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" event={"ID":"c0adb901-04ba-48f0-bd2e-f2d4a980b45a","Type":"ContainerDied","Data":"587f68060ec2b355c197f9cbc1bae194bbe445ea7fac31d79adc42a67e53a0ef"} Jul 07 17:57:41.312704 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.312555 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s" Jul 07 17:57:41.312704 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.312625 2569 scope.go:117] "RemoveContainer" containerID="159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df" Jul 07 17:57:41.322023 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.322005 2569 scope.go:117] "RemoveContainer" containerID="85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef" Jul 07 17:57:41.329887 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.329872 2569 scope.go:117] "RemoveContainer" containerID="159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df" Jul 07 17:57:41.330124 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:57:41.330104 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df\": container with ID starting with 159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df not found: ID does not exist" containerID="159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df" Jul 07 17:57:41.330163 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.330131 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df"} err="failed to get container status \"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df\": rpc error: code = NotFound desc = could not find container \"159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df\": container with ID starting with 159c74cce52136b279ea6838a6c47475465d10c6bc24977a258a88e43aae96df not found: ID does not exist" Jul 07 17:57:41.330163 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.330147 2569 scope.go:117] "RemoveContainer" containerID="85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef" Jul 07 17:57:41.330404 ip-10-0-137-197 kubenswrapper[2569]: E0707 17:57:41.330386 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef\": container with ID starting with 85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef not found: ID does not exist" containerID="85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef" Jul 07 17:57:41.330453 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.330411 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef"} err="failed to get container status \"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef\": rpc error: code = NotFound desc = could not find container \"85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef\": container with ID starting with 85f8f7324a11a820b618ab2a09ceb546d4652b5f7ba57518cfba224ae82257ef not found: ID does not exist" Jul 07 17:57:41.337567 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.337546 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:57:41.345587 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.345567 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb345660-openshift-default-85c44c67ff-26v7s"] Jul 07 17:57:41.451074 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:41.451042 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" path="/var/lib/kubelet/pods/c0adb901-04ba-48f0-bd2e-f2d4a980b45a/volumes" Jul 07 17:57:47.143554 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:47.143513 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:47.257246 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:47.257213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:47.916524 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:47.916457 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:52.776605 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:52.776563 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:57:54.133433 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:54.133402 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:54.255874 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:54.255839 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:54.354815 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:57:54.354782 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:05.512617 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:05.512584 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:06.111107 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:06.111020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:27.352980 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:27.352929 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:28.904008 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:28.903973 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:34.279184 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:34.279147 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:34.382225 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:34.382185 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:34.766178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:34.766139 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:35.161206 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161162 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 17:58:35.161637 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161617 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerName="maas-api" Jul 07 17:58:35.161737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161639 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerName="maas-api" Jul 07 17:58:35.161737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161651 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.161737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161660 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.161737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161690 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.161737 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161700 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.162000 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161802 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6647cb3f-8648-4c60-a0d6-1bdfb199de57" containerName="maas-api" Jul 07 17:58:35.162000 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161820 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.162000 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.161829 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0adb901-04ba-48f0-bd2e-f2d4a980b45a" containerName="istio-proxy" Jul 07 17:58:35.164874 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.164853 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.168531 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.168512 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f629b8-openshift-default-dockercfg-tccgd\"" Jul 07 17:58:35.180332 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.180275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:35.180444 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.180340 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 17:58:35.314877 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.314840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.314943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.314975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.315272 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.315135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416362 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416362 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416362 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416362 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416385 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416698 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416961 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416729 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.416961 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.417058 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416969 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.417058 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.416990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.417262 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.417102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.418303 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.418221 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:35.419176 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.419156 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.419277 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.419260 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.425906 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.425883 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.426262 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.426240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh\") pod \"e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.475399 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.475366 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:35.623307 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.623262 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 17:58:35.625043 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:58:35.625014 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5cfd68_e48f_4211_9edc_6fd73f2e9d56.slice/crio-a8790748118594b481763b9034e3c120c59bbb65d69315b66cbe1d860439c8a4 WatchSource:0}: Error finding container a8790748118594b481763b9034e3c120c59bbb65d69315b66cbe1d860439c8a4: Status 404 returned error can't find the container with id a8790748118594b481763b9034e3c120c59bbb65d69315b66cbe1d860439c8a4 Jul 07 17:58:35.627359 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.627323 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:35.627470 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.627404 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:35.627470 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:35.627451 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:36.571026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:36.570988 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerStarted","Data":"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c"} Jul 07 17:58:36.571665 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:36.571045 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerStarted","Data":"a8790748118594b481763b9034e3c120c59bbb65d69315b66cbe1d860439c8a4"} Jul 07 17:58:36.599950 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:36.599896 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podStartSLOduration=1.599881694 podStartE2EDuration="1.599881694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:58:36.59837133 +0000 UTC m=+2181.718575660" watchObservedRunningTime="2026-07-07 17:58:36.599881694 +0000 UTC m=+2181.720086084" Jul 07 17:58:37.475767 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:37.475730 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:37.480868 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:37.480842 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:37.575589 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:37.575556 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:37.576628 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:37.576606 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:38.792664 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.792626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:38.898910 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.898876 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:38.939243 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.939204 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 17:58:38.942820 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.942803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:38.947026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.946999 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:58:38.947026 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.947016 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-f629b8-serving-cert\"" Jul 07 17:58:38.961459 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.961434 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 17:58:38.999503 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:38.999470 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:39.056559 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.056470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.056559 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.056509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.056559 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.056534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5p5\" (UniqueName: \"kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.092146 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.092113 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:39.157377 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.157340 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.157546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.157391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.157546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.157418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5p5\" (UniqueName: \"kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.158028 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.158004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.160005 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.159981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.169239 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.169218 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5p5\" (UniqueName: \"kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5\") pod \"maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.253687 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.253647 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:39.419656 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.419633 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 17:58:39.421958 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:58:39.421933 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe93cc06_0604_43be_ae3e_6f169798ac6e.slice/crio-3fbdd3a46be8c6e87540c31151c88f76de316a75819877a3a661153564f95d0f WatchSource:0}: Error finding container 3fbdd3a46be8c6e87540c31151c88f76de316a75819877a3a661153564f95d0f: Status 404 returned error can't find the container with id 3fbdd3a46be8c6e87540c31151c88f76de316a75819877a3a661153564f95d0f Jul 07 17:58:39.426957 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.426933 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:58:39.521003 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.520972 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:39.586990 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.586902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" event={"ID":"be93cc06-0604-43be-ae3e-6f169798ac6e","Type":"ContainerStarted","Data":"3fbdd3a46be8c6e87540c31151c88f76de316a75819877a3a661153564f95d0f"} Jul 07 17:58:39.616732 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:39.616691 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:40.593480 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:40.593440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" event={"ID":"be93cc06-0604-43be-ae3e-6f169798ac6e","Type":"ContainerStarted","Data":"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec"} Jul 07 17:58:40.593885 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:40.593510 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:40.619094 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:40.619044 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" podStartSLOduration=2.039970643 podStartE2EDuration="2.619029024s" totalImagesPullingTime="579.058381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:58:40.616813713 +0000 UTC m=+2185.737018046" watchObservedRunningTime="2026-07-07 17:58:40.619029024 +0000 UTC m=+2185.739233365" Jul 07 17:58:43.313706 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:43.313670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:43.686960 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:43.686923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:44.062864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.062786 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 17:58:44.066531 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.066512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.069930 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.069910 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b6b8d-openshift-default-dockercfg-pj5qj\"" Jul 07 17:58:44.083333 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.083303 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 17:58:44.100665 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.100864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.100864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.100864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.100864 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.101091 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.101091 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.101091 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.101091 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.100935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7h6\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.115832 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.115797 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:44.201929 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.201892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202138 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.201945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202138 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.201984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202138 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202138 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7h6\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202138 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202459 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202459 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202459 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202627 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202488 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202753 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202706 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.202822 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.202744 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.203027 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.203005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.203096 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.203078 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.204572 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.204546 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.204691 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.204654 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.211637 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.211613 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.211846 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.211824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7h6\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6\") pod \"e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.378308 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.378186 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:44.534012 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.533980 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 17:58:44.535325 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:58:44.535276 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889a6f34_abdc_4948_81d4_7cffdd85cc4e.slice/crio-780aa5e784bcad5de3b4e89bf98b3a72888722f7444151abd5e0d9dad2e0f7cc WatchSource:0}: Error finding container 780aa5e784bcad5de3b4e89bf98b3a72888722f7444151abd5e0d9dad2e0f7cc: Status 404 returned error can't find the container with id 780aa5e784bcad5de3b4e89bf98b3a72888722f7444151abd5e0d9dad2e0f7cc Jul 07 17:58:44.537670 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.537634 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:44.537747 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.537708 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:44.537802 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.537762 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:44.613942 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.613908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerStarted","Data":"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb"} Jul 07 17:58:44.614059 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.613970 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerStarted","Data":"780aa5e784bcad5de3b4e89bf98b3a72888722f7444151abd5e0d9dad2e0f7cc"} Jul 07 17:58:44.633619 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.633544 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:44.650103 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:44.650048 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podStartSLOduration=0.650027265 podStartE2EDuration="650.027265ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:58:44.648495464 +0000 UTC m=+2189.768699795" watchObservedRunningTime="2026-07-07 17:58:44.650027265 +0000 UTC m=+2189.770231594" Jul 07 17:58:45.378703 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:45.378665 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:45.383848 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:45.383823 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:45.618417 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:45.618384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:45.619616 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:45.619596 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:58:46.604983 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:46.604955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 17:58:46.658370 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:46.658333 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:46.784158 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:46.784123 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:46.843824 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:46.843786 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:58:46.881122 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:46.881030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:47.263505 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:47.263453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:47.648788 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:47.648761 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:47.749527 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:47.749491 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:51.225161 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:51.225124 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:53.616485 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:53.616437 2569 patch_prober.go:28] interesting pod/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 17:58:53.616994 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:53.616498 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 17:58:54.664262 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.664225 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerID="9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c" exitCode=0 Jul 07 17:58:54.664713 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.664318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerDied","Data":"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c"} Jul 07 17:58:54.664771 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.664738 2569 scope.go:117] "RemoveContainer" containerID="9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c" Jul 07 17:58:54.665830 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.665802 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:54.665888 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.665873 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:54.665931 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:54.665905 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:58:55.671325 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:55.671275 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerStarted","Data":"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6"} Jul 07 17:58:56.475816 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:56.475776 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:58:56.476009 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:56.475965 2569 patch_prober.go:28] interesting pod/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 07 17:58:56.476069 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:56.476046 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 07 17:58:57.476966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:57.476523 2569 patch_prober.go:28] interesting pod/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 07 17:58:57.476966 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:57.476609 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 07 17:58:57.485404 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:57.485375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:58:58.475987 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:58.475948 2569 patch_prober.go:28] interesting pod/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 07 17:58:58.476157 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:58.476014 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 07 17:58:59.476302 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:59.476261 2569 patch_prober.go:28] interesting pod/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 07 17:58:59.476720 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:58:59.476341 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 07 17:59:00.373230 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:00.373186 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:00.473546 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:00.473513 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:00.479033 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:00.479009 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:59:00.479498 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:00.479392 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:59:00.480189 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:00.480171 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 17:59:01.417440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.416982 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:37510->10.132.0.50:15021: read: connection reset by peer" start-of-body= Jul 07 17:59:01.417440 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.417046 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:37510->10.132.0.50:15021: read: connection reset by peer" Jul 07 17:59:01.702685 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.702603 2569 generic.go:356] "Generic (PLEG): container finished" podID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerID="83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb" exitCode=0 Jul 07 17:59:01.703041 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.702681 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerDied","Data":"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb"} Jul 07 17:59:01.703227 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.703209 2569 scope.go:117] "RemoveContainer" containerID="83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb" Jul 07 17:59:01.704228 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.704199 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:59:01.704328 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.704309 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:59:01.704391 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:01.704377 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 17:59:02.711076 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:02.711035 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerStarted","Data":"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1"} Jul 07 17:59:03.378910 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:03.378870 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:59:03.379154 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:03.379132 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 07 17:59:03.379217 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:03.379181 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 07 17:59:04.379650 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:04.379613 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 07 17:59:04.380074 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:04.379673 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 07 17:59:05.378911 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:05.378870 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 07 17:59:05.379105 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:05.378931 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 07 17:59:06.379469 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:06.379430 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 07 17:59:06.379917 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:06.379485 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 07 17:59:07.382588 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:07.382554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:59:07.382978 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:07.382849 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:59:07.383640 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:07.383618 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 17:59:08.621593 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:08.621556 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:08.727409 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:08.727374 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:13.563245 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:13.563202 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:15.992178 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:15.992141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:59:29.699480 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:29.699444 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:59:33.822269 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:33.822231 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:40.398510 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.398416 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 17:59:40.402324 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.402275 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.412038 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.412012 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b6b8d\"/\"kube-root-ca.crt\"" Jul 07 17:59:40.412157 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.412012 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b6b8d\"/\"default-dockercfg-mhz8l\"" Jul 07 17:59:40.412157 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.412020 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b6b8d\"/\"test-model-7b6b8d-kserve-self-signed-certs\"" Jul 07 17:59:40.412301 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.412019 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b6b8d\"/\"openshift-service-ca.crt\"" Jul 07 17:59:40.420313 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.420265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 17:59:40.436547 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.436517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:40.566841 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.567036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.567036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.567036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv294\" (UniqueName: \"kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.567036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.567036 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.566985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668552 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv294\" (UniqueName: \"kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668817 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668817 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.668817 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.669012 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.668990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.669083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.669036 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.669083 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.669070 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.670909 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.670888 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.671084 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.671067 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.681455 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.681431 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv294\" (UniqueName: \"kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294\") pod \"test-model-7b6b8d-kserve-86b9cb9659-sr5l7\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.711143 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.711109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:40.862080 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.862051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 17:59:40.864375 ip-10-0-137-197 kubenswrapper[2569]: W0707 17:59:40.864349 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode00c21d6_7aff_41b3_a073_66d8930ff018.slice/crio-64dfc570b4789b1e93c92172ae2c7a2dd109e49fc7a9f47b4d91ff36c796ee82 WatchSource:0}: Error finding container 64dfc570b4789b1e93c92172ae2c7a2dd109e49fc7a9f47b4d91ff36c796ee82: Status 404 returned error can't find the container with id 64dfc570b4789b1e93c92172ae2c7a2dd109e49fc7a9f47b4d91ff36c796ee82 Jul 07 17:59:40.906604 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:40.906572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" event={"ID":"e00c21d6-7aff-41b3-a073-66d8930ff018","Type":"ContainerStarted","Data":"64dfc570b4789b1e93c92172ae2c7a2dd109e49fc7a9f47b4d91ff36c796ee82"} Jul 07 17:59:41.912730 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:41.912696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" event={"ID":"e00c21d6-7aff-41b3-a073-66d8930ff018","Type":"ContainerStarted","Data":"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f"} Jul 07 17:59:44.919240 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:44.919178 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" podStartSLOduration=4.919159836 podStartE2EDuration="4.919159836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:59:41.94175748 +0000 UTC m=+2247.061961810" watchObservedRunningTime="2026-07-07 17:59:44.919159836 +0000 UTC m=+2250.039364165" Jul 07 17:59:44.919826 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:44.919803 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 17:59:50.711719 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:50.711688 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:50.712142 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:50.711739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:50.724207 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:50.724180 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:50.968329 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:50.968226 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 17:59:59.405480 ip-10-0-137-197 kubenswrapper[2569]: I0707 17:59:59.405442 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:00:00.982664 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:00.982633 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:02.075821 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.075785 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 18:00:02.157055 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.157020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:02.332709 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.332603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:02.705682 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.705648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:02.830571 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.830533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:02.927241 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:02.927209 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:03.116450 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.116364 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 18:00:03.116836 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.116568 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" podUID="be93cc06-0604-43be-ae3e-6f169798ac6e" containerName="maas-api" containerID="cri-o://22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec" gracePeriod=30 Jul 07 18:00:03.221192 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.221160 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:03.367043 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.366977 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 18:00:03.458024 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.458000 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk5p5\" (UniqueName: \"kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5\") pod \"be93cc06-0604-43be-ae3e-6f169798ac6e\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " Jul 07 18:00:03.458203 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.458055 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls\") pod \"be93cc06-0604-43be-ae3e-6f169798ac6e\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " Jul 07 18:00:03.458203 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.458121 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca\") pod \"be93cc06-0604-43be-ae3e-6f169798ac6e\" (UID: \"be93cc06-0604-43be-ae3e-6f169798ac6e\") " Jul 07 18:00:03.458563 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.458532 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be93cc06-0604-43be-ae3e-6f169798ac6e" (UID: "be93cc06-0604-43be-ae3e-6f169798ac6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:00:03.460277 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.460258 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5" (OuterVolumeSpecName: "kube-api-access-tk5p5") pod "be93cc06-0604-43be-ae3e-6f169798ac6e" (UID: "be93cc06-0604-43be-ae3e-6f169798ac6e"). InnerVolumeSpecName "kube-api-access-tk5p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:03.460517 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.460494 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be93cc06-0604-43be-ae3e-6f169798ac6e" (UID: "be93cc06-0604-43be-ae3e-6f169798ac6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:00:03.559492 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.559457 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be93cc06-0604-43be-ae3e-6f169798ac6e-openshift-service-ca\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:03.559492 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.559488 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5p5\" (UniqueName: \"kubernetes.io/projected/be93cc06-0604-43be-ae3e-6f169798ac6e-kube-api-access-tk5p5\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:03.559492 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:03.559499 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be93cc06-0604-43be-ae3e-6f169798ac6e-maas-api-tls\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:04.021116 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.021083 2569 generic.go:356] "Generic (PLEG): container finished" podID="be93cc06-0604-43be-ae3e-6f169798ac6e" containerID="22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec" exitCode=0 Jul 07 18:00:04.021326 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.021128 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" Jul 07 18:00:04.021326 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.021167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" event={"ID":"be93cc06-0604-43be-ae3e-6f169798ac6e","Type":"ContainerDied","Data":"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec"} Jul 07 18:00:04.021326 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.021209 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42" event={"ID":"be93cc06-0604-43be-ae3e-6f169798ac6e","Type":"ContainerDied","Data":"3fbdd3a46be8c6e87540c31151c88f76de316a75819877a3a661153564f95d0f"} Jul 07 18:00:04.021326 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.021229 2569 scope.go:117] "RemoveContainer" containerID="22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec" Jul 07 18:00:04.030542 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.030525 2569 scope.go:117] "RemoveContainer" containerID="22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec" Jul 07 18:00:04.030782 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:00:04.030761 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec\": container with ID starting with 22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec not found: ID does not exist" containerID="22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec" Jul 07 18:00:04.030871 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.030790 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec"} err="failed to get container status \"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec\": rpc error: code = NotFound desc = could not find container \"22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec\": container with ID starting with 22f02cae85e7ff89e1c3cb1eb910cf48f1b047f7308c30513d6ddca1919ab0ec not found: ID does not exist" Jul 07 18:00:04.045792 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.045757 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 18:00:04.055178 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:04.049229 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-f629b8-7d5455c4ff-2vz42"] Jul 07 18:00:05.451410 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:05.451378 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be93cc06-0604-43be-ae3e-6f169798ac6e" path="/var/lib/kubelet/pods/be93cc06-0604-43be-ae3e-6f169798ac6e/volumes" Jul 07 18:00:07.748994 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:07.748958 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:14.291059 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:14.291022 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:00:17.097232 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.097190 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" podUID="e00c21d6-7aff-41b3-a073-66d8930ff018" containerName="main" containerID="cri-o://277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f" gracePeriod=45 Jul 07 18:00:17.356151 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.356084 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 18:00:17.385872 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.385807 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv294\" (UniqueName: \"kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386063 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.385882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386063 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.385918 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386063 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.385949 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386063 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386017 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386063 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386054 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm\") pod \"e00c21d6-7aff-41b3-a073-66d8930ff018\" (UID: \"e00c21d6-7aff-41b3-a073-66d8930ff018\") " Jul 07 18:00:17.386376 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386270 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:17.386376 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386348 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache" (OuterVolumeSpecName: "model-cache") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:17.386488 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386452 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-model-cache\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.386544 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386493 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-tmp-dir\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.386935 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.386912 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home" (OuterVolumeSpecName: "home") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:17.388394 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.388368 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294" (OuterVolumeSpecName: "kube-api-access-xv294") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "kube-api-access-xv294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:17.388858 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.388833 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm" (OuterVolumeSpecName: "dshm") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:17.388949 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.388863 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e00c21d6-7aff-41b3-a073-66d8930ff018" (UID: "e00c21d6-7aff-41b3-a073-66d8930ff018"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:00:17.487398 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.487364 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv294\" (UniqueName: \"kubernetes.io/projected/e00c21d6-7aff-41b3-a073-66d8930ff018-kube-api-access-xv294\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.487398 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.487394 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c21d6-7aff-41b3-a073-66d8930ff018-tls-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.487398 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.487403 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-home\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.487677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:17.487420 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c21d6-7aff-41b3-a073-66d8930ff018-dshm\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:18.090502 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.090462 2569 generic.go:356] "Generic (PLEG): container finished" podID="e00c21d6-7aff-41b3-a073-66d8930ff018" containerID="277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f" exitCode=0 Jul 07 18:00:18.090682 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.090506 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" Jul 07 18:00:18.090682 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.090543 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" event={"ID":"e00c21d6-7aff-41b3-a073-66d8930ff018","Type":"ContainerDied","Data":"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f"} Jul 07 18:00:18.090682 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.090580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7" event={"ID":"e00c21d6-7aff-41b3-a073-66d8930ff018","Type":"ContainerDied","Data":"64dfc570b4789b1e93c92172ae2c7a2dd109e49fc7a9f47b4d91ff36c796ee82"} Jul 07 18:00:18.090682 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.090601 2569 scope.go:117] "RemoveContainer" containerID="277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f" Jul 07 18:00:18.099769 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.099543 2569 scope.go:117] "RemoveContainer" containerID="277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f" Jul 07 18:00:18.100100 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:00:18.099858 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f\": container with ID starting with 277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f not found: ID does not exist" containerID="277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f" Jul 07 18:00:18.100100 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.099894 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f"} err="failed to get container status \"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f\": rpc error: code = NotFound desc = could not find container \"277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f\": container with ID starting with 277f215f5671687c726b37092c1fee78fda055ab6c7b907f6c1ac0262699096f not found: ID does not exist" Jul 07 18:00:18.112389 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.112363 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 18:00:18.117617 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:18.117591 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7b6b8d/test-model-7b6b8d-kserve-86b9cb9659-sr5l7"] Jul 07 18:00:19.451502 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:19.451471 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00c21d6-7aff-41b3-a073-66d8930ff018" path="/var/lib/kubelet/pods/e00c21d6-7aff-41b3-a073-66d8930ff018/volumes" Jul 07 18:00:42.933470 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:42.933434 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:43.039047 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:43.039015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:47.413804 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:47.413754 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:47.750216 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:47.750127 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 18:00:47.750397 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:47.750375 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" containerID="cri-o://344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6" gracePeriod=30 Jul 07 18:00:47.785625 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:47.785591 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:48.097457 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:48.097356 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:48.483512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:48.483477 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:48.884740 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:48.884656 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:49.296499 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:49.296460 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:49.393245 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:49.393206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:49.673678 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:49.673642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:52.247553 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:52.247519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:00:53.016278 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.016211 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 18:00:53.122153 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122118 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122372 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122179 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122372 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122372 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122258 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122526 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122413 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122526 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122456 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122526 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122517 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122693 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122519 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:53.122693 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122551 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122693 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket\") pod \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\" (UID: \"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56\") " Jul 07 18:00:53.122857 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122694 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:00:53.122857 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122771 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:53.122961 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122951 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.123017 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122969 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.123017 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122985 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.123017 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.122951 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:53.123361 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.123123 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:53.124774 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.124747 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:53.124902 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.124844 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:53.124902 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.124858 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh" (OuterVolumeSpecName: "kube-api-access-qdfxh") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "kube-api-access-qdfxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:53.124993 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.124950 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" (UID: "3d5cfd68-e48f-4211-9edc-6fd73f2e9d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:00:53.223955 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223913 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-kube-api-access-qdfxh\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.223955 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223947 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.223955 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223958 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.223955 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223967 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.224235 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223975 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.224235 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.223983 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:00:53.261742 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.261711 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerID="344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6" exitCode=0 Jul 07 18:00:53.262121 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.261762 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" Jul 07 18:00:53.262121 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.261793 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerDied","Data":"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6"} Jul 07 18:00:53.262121 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.261854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw" event={"ID":"3d5cfd68-e48f-4211-9edc-6fd73f2e9d56","Type":"ContainerDied","Data":"a8790748118594b481763b9034e3c120c59bbb65d69315b66cbe1d860439c8a4"} Jul 07 18:00:53.262121 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.261885 2569 scope.go:117] "RemoveContainer" containerID="344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6" Jul 07 18:00:53.271548 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.271530 2569 scope.go:117] "RemoveContainer" containerID="9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c" Jul 07 18:00:53.280238 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.280216 2569 scope.go:117] "RemoveContainer" containerID="344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6" Jul 07 18:00:53.280623 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:00:53.280595 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6\": container with ID starting with 344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6 not found: ID does not exist" containerID="344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6" Jul 07 18:00:53.280711 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.280634 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6"} err="failed to get container status \"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6\": rpc error: code = NotFound desc = could not find container \"344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6\": container with ID starting with 344b03d54484345ee5433b7f8af21f69b72f330e2709a76bd5fba490ac40abf6 not found: ID does not exist" Jul 07 18:00:53.280711 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.280660 2569 scope.go:117] "RemoveContainer" containerID="9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c" Jul 07 18:00:53.281102 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:00:53.281077 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c\": container with ID starting with 9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c not found: ID does not exist" containerID="9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c" Jul 07 18:00:53.281186 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.281110 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c"} err="failed to get container status \"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c\": rpc error: code = NotFound desc = could not find container \"9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c\": container with ID starting with 9bf462f8ebf619716c8d35c34ec1981c83e16b4067a318f2f6ec1c020bbef53c not found: ID does not exist" Jul 07 18:00:53.295883 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.295852 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 18:00:53.307085 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.307053 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f629b8-openshift-default-b6c487675-kt5fw"] Jul 07 18:00:53.452597 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:53.452551 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" path="/var/lib/kubelet/pods/3d5cfd68-e48f-4211-9edc-6fd73f2e9d56/volumes" Jul 07 18:00:57.169190 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.169154 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:57.292829 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.292799 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:57.545415 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.545315 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 18:00:57.545609 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.545563 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" containerID="cri-o://de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1" gracePeriod=30 Jul 07 18:00:57.563588 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.563560 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:57.801121 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:57.801035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:58.196022 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.195992 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:58.588743 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.588661 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:00:58.589118 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589100 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00c21d6-7aff-41b3-a073-66d8930ff018" containerName="main" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589122 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00c21d6-7aff-41b3-a073-66d8930ff018" containerName="main" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589139 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be93cc06-0604-43be-ae3e-6f169798ac6e" containerName="maas-api" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589147 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be93cc06-0604-43be-ae3e-6f169798ac6e" containerName="maas-api" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589159 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589168 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589181 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589189 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589298 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589313 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5cfd68-e48f-4211-9edc-6fd73f2e9d56" containerName="istio-proxy" Jul 07 18:00:58.589589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589329 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00c21d6-7aff-41b3-a073-66d8930ff018" containerName="main" Jul 07 18:00:58.589589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.589339 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be93cc06-0604-43be-ae3e-6f169798ac6e" containerName="maas-api" Jul 07 18:00:58.593900 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.593876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.594061 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.593959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:58.602459 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.602427 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-17ec16-openshift-default-dockercfg-b4fl8\"" Jul 07 18:00:58.613500 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.613472 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:00:58.673954 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.673912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.673954 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.673959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674184 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.673981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674184 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674184 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674305 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674305 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674374 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6nh\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.674409 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.674379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775081 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6nh\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775607 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775340 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775607 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775607 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775607 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775607 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775558 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.775865 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.775844 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.776076 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.776050 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.776202 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.776139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.777710 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.777680 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.777801 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.777733 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.791999 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.791969 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6nh\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.799472 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.799438 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token\") pod \"e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:58.906463 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:58.906429 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:00:59.071709 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.071672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:00:59.076083 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:00:59.076057 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86da86a5_2b5d_41b7_a5a9_74cf378d4e66.slice/crio-268ae17f69403af499677ad0db620c20fa704984576300aba3fcfed591ac61af WatchSource:0}: Error finding container 268ae17f69403af499677ad0db620c20fa704984576300aba3fcfed591ac61af: Status 404 returned error can't find the container with id 268ae17f69403af499677ad0db620c20fa704984576300aba3fcfed591ac61af Jul 07 18:00:59.078182 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.078135 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:00:59.078265 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.078227 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:00:59.078330 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.078271 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:00:59.294969 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.294864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerStarted","Data":"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8"} Jul 07 18:00:59.294969 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.294923 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerStarted","Data":"268ae17f69403af499677ad0db620c20fa704984576300aba3fcfed591ac61af"} Jul 07 18:00:59.364463 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.364409 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podStartSLOduration=1.364394131 podStartE2EDuration="1.364394131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:59.362519836 +0000 UTC m=+2324.482724178" watchObservedRunningTime="2026-07-07 18:00:59.364394131 +0000 UTC m=+2324.484598460" Jul 07 18:00:59.365543 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.365518 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:59.906896 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:00:59.906854 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:00.912813 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:00.912779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:01.305015 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:01.304915 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:01.306131 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:01.306106 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:01.630637 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:01.630541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:01.814430 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:01.814398 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:02.013317 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.013263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:02.515413 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.515366 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:02.795580 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.795549 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 18:01:02.913597 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913564 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913604 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s7h6\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913635 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913723 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.913780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913752 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913808 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913833 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy\") pod \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\" (UID: \"889a6f34-abdc-4948-81d4-7cffdd85cc4e\") " Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.913892 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.914043 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.914209 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:01:02.914231 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.914226 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:02.915451 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.914248 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:02.915451 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.914633 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:02.915451 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.915090 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:02.916427 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.916403 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6" (OuterVolumeSpecName: "kube-api-access-4s7h6") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "kube-api-access-4s7h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:02.916740 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.916719 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:01:02.916868 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.916848 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:02.916868 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:02.916857 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "889a6f34-abdc-4948-81d4-7cffdd85cc4e" (UID: "889a6f34-abdc-4948-81d4-7cffdd85cc4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:03.015150 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015106 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7h6\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-kube-api-access-4s7h6\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015150 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015143 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015150 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015156 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015646 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015167 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015646 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015179 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015646 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015189 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.015646 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.015200 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889a6f34-abdc-4948-81d4-7cffdd85cc4e-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:03.019568 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.019539 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:03.182276 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.182240 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:03.317585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.317544 2569 generic.go:356] "Generic (PLEG): container finished" podID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerID="de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1" exitCode=0 Jul 07 18:01:03.317585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.317582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerDied","Data":"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1"} Jul 07 18:01:03.317821 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.317616 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" Jul 07 18:01:03.317821 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.317678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn" event={"ID":"889a6f34-abdc-4948-81d4-7cffdd85cc4e","Type":"ContainerDied","Data":"780aa5e784bcad5de3b4e89bf98b3a72888722f7444151abd5e0d9dad2e0f7cc"} Jul 07 18:01:03.318937 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.317702 2569 scope.go:117] "RemoveContainer" containerID="de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1" Jul 07 18:01:03.328351 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.328332 2569 scope.go:117] "RemoveContainer" containerID="83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb" Jul 07 18:01:03.337147 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.337129 2569 scope.go:117] "RemoveContainer" containerID="de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1" Jul 07 18:01:03.337671 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:03.337637 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1\": container with ID starting with de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1 not found: ID does not exist" containerID="de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1" Jul 07 18:01:03.337671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.337683 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1"} err="failed to get container status \"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1\": rpc error: code = NotFound desc = could not find container \"de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1\": container with ID starting with de2f7836fc4a29bb2628fdbb4e5cb1df82259831d5bd6642a87a0d0adb27c7e1 not found: ID does not exist" Jul 07 18:01:03.337671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.337704 2569 scope.go:117] "RemoveContainer" containerID="83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb" Jul 07 18:01:03.338008 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:03.337988 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb\": container with ID starting with 83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb not found: ID does not exist" containerID="83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb" Jul 07 18:01:03.338073 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.338012 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb"} err="failed to get container status \"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb\": rpc error: code = NotFound desc = could not find container \"83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb\": container with ID starting with 83abad7f9e680a2514adfafea6f12e890f0c21616658e2b7f46c7ee33fd2d7fb not found: ID does not exist" Jul 07 18:01:03.365883 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.365847 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 18:01:03.368049 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.368023 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b6b8d-openshift-default-7bc69c7657-wq7gn"] Jul 07 18:01:03.451418 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:03.451336 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" path="/var/lib/kubelet/pods/889a6f34-abdc-4948-81d4-7cffdd85cc4e/volumes" Jul 07 18:01:06.127255 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:06.127170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:06.664586 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:06.664552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:07.117538 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.117446 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:07.117910 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.117890 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.118012 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.117917 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.118012 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.117933 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.118012 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.117941 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.118166 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.118061 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.118166 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.118073 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="889a6f34-abdc-4948-81d4-7cffdd85cc4e" containerName="istio-proxy" Jul 07 18:01:07.122391 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.122369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.128736 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.128709 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-97c1e9-openshift-default-dockercfg-5ll9c\"" Jul 07 18:01:07.138446 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.138417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:07.156781 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.156748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:07.256626 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.256815 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.256815 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.256815 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.256932 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.256932 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8qq\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.257000 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256939 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.257000 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.256980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.257068 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.257002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.354554 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.351961 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:01:07.358229 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358373 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358373 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358373 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8qq\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358433 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358725 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358725 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358838 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358765 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.358902 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358862 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.359025 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.358998 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.359152 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.359135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.360728 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.360703 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.361234 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.361213 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.374743 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.374671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8qq\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.375710 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.375689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token\") pod \"e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.433480 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.433439 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:07.493821 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.493714 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:07.599395 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.599371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:07.601492 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:01:07.601465 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db2e041_9041_4ff6_a026_b0274958d058.slice/crio-528d8494cd5a5b277acf6c769e86ce2e65fd0226768a5cc3038c32fcdac8f844 WatchSource:0}: Error finding container 528d8494cd5a5b277acf6c769e86ce2e65fd0226768a5cc3038c32fcdac8f844: Status 404 returned error can't find the container with id 528d8494cd5a5b277acf6c769e86ce2e65fd0226768a5cc3038c32fcdac8f844 Jul 07 18:01:07.603639 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.603599 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:07.603716 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.603676 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:07.603716 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:07.603705 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:08.347476 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:08.347439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerStarted","Data":"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875"} Jul 07 18:01:08.347890 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:08.347502 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerStarted","Data":"528d8494cd5a5b277acf6c769e86ce2e65fd0226768a5cc3038c32fcdac8f844"} Jul 07 18:01:08.382396 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:08.382344 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podStartSLOduration=1.3823292010000001 podStartE2EDuration="1.382329201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:08.380563122 +0000 UTC m=+2333.500767451" watchObservedRunningTime="2026-07-07 18:01:08.382329201 +0000 UTC m=+2333.502533577" Jul 07 18:01:08.433832 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:08.433789 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:08.439221 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:08.439194 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:09.352393 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:09.352357 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:09.353541 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:09.353523 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:10.540217 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:10.540156 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:10.650673 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:10.650624 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:10.879927 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:10.879804 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:11.502415 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:11.502372 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:12.357899 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:12.357865 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:12.496997 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:12.496958 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:15.086227 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.086193 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:15.384428 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.384331 2569 generic.go:356] "Generic (PLEG): container finished" podID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerID="80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8" exitCode=0 Jul 07 18:01:15.384428 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.384408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerDied","Data":"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8"} Jul 07 18:01:15.384842 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.384828 2569 scope.go:117] "RemoveContainer" containerID="80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8" Jul 07 18:01:15.385878 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.385850 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:15.385937 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.385922 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:15.385971 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:15.385953 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:16.304881 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:16.304831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:16.391512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:16.391473 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerStarted","Data":"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb"} Jul 07 18:01:16.907888 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:16.907845 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:16.909174 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:16.909135 2569 patch_prober.go:28] interesting pod/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 18:01:16.909360 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:16.909198 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 18:01:17.907791 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:17.907741 2569 patch_prober.go:28] interesting pod/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 18:01:17.908411 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:17.907816 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 18:01:18.908033 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:18.907994 2569 patch_prober.go:28] interesting pod/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 18:01:18.909130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:18.908662 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 18:01:19.907696 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:19.907649 2569 patch_prober.go:28] interesting pod/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 07 18:01:19.908029 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:19.907721 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 07 18:01:20.911572 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:20.911541 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:20.912025 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:20.911948 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:20.912791 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:20.912763 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:22.610316 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:22.610257 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:01:23.507494 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:23.507457 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:24.084048 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.083942 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:24.266393 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.266349 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:24.354633 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.354545 2569 patch_prober.go:28] interesting pod/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 07 18:01:24.354785 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.354624 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 07 18:01:24.376698 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.376661 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:24.445671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.445641 2569 generic.go:356] "Generic (PLEG): container finished" podID="1db2e041-9041-4ff6-a026-b0274958d058" containerID="08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875" exitCode=0 Jul 07 18:01:24.445851 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.445721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerDied","Data":"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875"} Jul 07 18:01:24.446080 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.446063 2569 scope.go:117] "RemoveContainer" containerID="08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875" Jul 07 18:01:24.447017 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.446990 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:24.447106 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.447055 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:24.447106 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.447087 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:24.615712 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:24.615603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:25.452595 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:25.452563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerStarted","Data":"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0"} Jul 07 18:01:26.434657 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:26.434613 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:26.434900 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:26.434876 2569 patch_prober.go:28] interesting pod/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 07 18:01:26.434965 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:26.434944 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 07 18:01:27.434332 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:27.434265 2569 patch_prober.go:28] interesting pod/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 07 18:01:27.434740 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:27.434356 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 07 18:01:28.434454 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:28.434416 2569 patch_prober.go:28] interesting pod/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 07 18:01:28.434848 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:28.434476 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 07 18:01:29.434465 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:29.434428 2569 patch_prober.go:28] interesting pod/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 07 18:01:29.434902 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:29.434494 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 07 18:01:30.437757 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:30.437723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:30.438139 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:30.437984 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:30.438721 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:30.438707 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:31.097303 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.097254 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:31.197034 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.196989 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:31.435596 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.435559 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:01:31.435807 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.435779 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" containerID="cri-o://630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb" gracePeriod=30 Jul 07 18:01:31.463786 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.463749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:31.692227 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.692139 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:31.969133 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:31.969038 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:32.337877 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:32.337786 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:32.548024 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:32.547981 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:32.644482 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:32.644452 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:32.884851 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:32.884817 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:35.016179 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:35.016141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:01:35.914708 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:35.914669 2569 patch_prober.go:28] interesting pod/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:01:35.914877 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:35.914746 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:01:36.693346 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.693318 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:36.748061 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.747974 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748061 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748014 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748061 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748061 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748061 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748114 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748147 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748180 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6nh\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748242 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748270 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy\") pod \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\" (UID: \"86da86a5-2b5d-41b7-a5a9-74cf378d4e66\") " Jul 07 18:01:36.748416 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748402 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:36.748720 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748655 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.748821 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.748796 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:01:36.749070 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.749027 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:36.749266 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.749213 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:36.749379 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.749310 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data" (OuterVolumeSpecName: "istio-data") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:36.750692 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.750663 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token" (OuterVolumeSpecName: "istio-token") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:36.750778 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.750758 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:36.751072 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.751052 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh" (OuterVolumeSpecName: "kube-api-access-gn6nh") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "kube-api-access-gn6nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:36.751203 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.751185 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86da86a5-2b5d-41b7-a5a9-74cf378d4e66" (UID: "86da86a5-2b5d-41b7-a5a9-74cf378d4e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:01:36.850085 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850025 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850085 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850077 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850085 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850089 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850085 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850099 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850447 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850112 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850447 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850123 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850447 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850136 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:36.850447 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:36.850149 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6nh\" (UniqueName: \"kubernetes.io/projected/86da86a5-2b5d-41b7-a5a9-74cf378d4e66-kube-api-access-gn6nh\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:37.512861 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.512821 2569 generic.go:356] "Generic (PLEG): container finished" podID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerID="630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb" exitCode=0 Jul 07 18:01:37.513041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.512876 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" Jul 07 18:01:37.513041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.512912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerDied","Data":"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb"} Jul 07 18:01:37.513041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.512973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm" event={"ID":"86da86a5-2b5d-41b7-a5a9-74cf378d4e66","Type":"ContainerDied","Data":"268ae17f69403af499677ad0db620c20fa704984576300aba3fcfed591ac61af"} Jul 07 18:01:37.513041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.513005 2569 scope.go:117] "RemoveContainer" containerID="630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb" Jul 07 18:01:37.522306 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.522260 2569 scope.go:117] "RemoveContainer" containerID="80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8" Jul 07 18:01:37.530438 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.530414 2569 scope.go:117] "RemoveContainer" containerID="630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb" Jul 07 18:01:37.530693 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:37.530668 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb\": container with ID starting with 630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb not found: ID does not exist" containerID="630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb" Jul 07 18:01:37.530793 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.530707 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb"} err="failed to get container status \"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb\": rpc error: code = NotFound desc = could not find container \"630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb\": container with ID starting with 630199ea0849ca627b86bb49a4dfb4bf9014c4e6935ae31fd0556a2bc3452bbb not found: ID does not exist" Jul 07 18:01:37.530793 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.530729 2569 scope.go:117] "RemoveContainer" containerID="80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8" Jul 07 18:01:37.530977 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:37.530957 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8\": container with ID starting with 80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8 not found: ID does not exist" containerID="80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8" Jul 07 18:01:37.531017 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.530983 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8"} err="failed to get container status \"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8\": rpc error: code = NotFound desc = could not find container \"80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8\": container with ID starting with 80048aebcd3f6d8a82fc80d761b17186d97bcd42bbe5ce6c304cf501698161b8 not found: ID does not exist" Jul 07 18:01:37.546431 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.546407 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:01:37.550122 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:37.550099 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-17ec16-openshift-default-7d95445c69-zp4zm"] Jul 07 18:01:39.451716 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.451684 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" path="/var/lib/kubelet/pods/86da86a5-2b5d-41b7-a5a9-74cf378d4e66/volumes" Jul 07 18:01:39.710433 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.710345 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:39.813125 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.813093 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:39.981159 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.981068 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:39.981364 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.981338 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" containerID="cri-o://2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0" gracePeriod=30 Jul 07 18:01:39.992303 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:39.992263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:40.242316 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.242196 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:40.630827 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.630745 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:40.940122 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.940084 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:01:40.940481 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.940468 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:40.940549 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.940483 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:40.940587 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.940551 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:40.940587 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.940559 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:40.943573 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.943556 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.951471 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.951451 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1e464b-openshift-default-dockercfg-htkn5\"" Jul 07 18:01:40.965548 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.965525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:01:40.977349 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.977322 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:40.990504 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990619 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990619 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990574 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990742 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990742 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990742 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990727 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990910 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990910 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7csr\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:40.990910 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:40.990784 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092182 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092391 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092391 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092488 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092488 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092689 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.092689 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092624 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.092597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.093140 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7csr\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.093357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.093384 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.093681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.096307 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.096171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.102347 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.102319 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.107409 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.107381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7csr\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.107525 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.107440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token\") pod \"e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.253741 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.253656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:41.321197 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.321113 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:41.418133 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.418103 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:01:41.419417 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:01:41.419391 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2caf4_c8bb_4ea6_8234_89badf791bc5.slice/crio-e93d123fb8b699398365d6ee21b308ee6a6c0a3a349cc25d69ec2c6659d4325e WatchSource:0}: Error finding container e93d123fb8b699398365d6ee21b308ee6a6c0a3a349cc25d69ec2c6659d4325e: Status 404 returned error can't find the container with id e93d123fb8b699398365d6ee21b308ee6a6c0a3a349cc25d69ec2c6659d4325e Jul 07 18:01:41.421693 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.421659 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:41.421763 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.421725 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:41.421763 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.421751 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:41.535860 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.535773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerStarted","Data":"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd"} Jul 07 18:01:41.535860 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.535838 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerStarted","Data":"e93d123fb8b699398365d6ee21b308ee6a6c0a3a349cc25d69ec2c6659d4325e"} Jul 07 18:01:41.587140 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:41.587078 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podStartSLOduration=1.587062397 podStartE2EDuration="1.587062397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:41.581232522 +0000 UTC m=+2366.701436853" watchObservedRunningTime="2026-07-07 18:01:41.587062397 +0000 UTC m=+2366.707266727" Jul 07 18:01:42.253984 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:42.253946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:42.259010 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:42.258982 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:42.540885 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:42.540785 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:42.541763 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:42.541744 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:43.731346 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:43.731265 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:43.874757 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:43.874717 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:43.990294 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:43.990181 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:44.334761 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:44.334672 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:44.641379 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:44.641354 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:44.742052 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:44.742007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:44.849272 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:44.849239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:44.948213 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:44.948116 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:45.237995 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.237968 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:45.335601 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335562 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k8qq\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335601 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335607 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335901 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335638 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335901 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335901 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335700 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335901 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335767 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.335901 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335820 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.336156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335975 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:45.336156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.335995 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:45.336156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336035 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.336156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336079 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket\") pod \"1db2e041-9041-4ff6-a026-b0274958d058\" (UID: \"1db2e041-9041-4ff6-a026-b0274958d058\") " Jul 07 18:01:45.336383 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336355 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:45.336383 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336262 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:01:45.336696 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336640 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db2e041-9041-4ff6-a026-b0274958d058-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.336696 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336661 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.336696 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336676 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.336696 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336691 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.337045 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.336982 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data" (OuterVolumeSpecName: "istio-data") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:45.338126 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.338096 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq" (OuterVolumeSpecName: "kube-api-access-8k8qq") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "kube-api-access-8k8qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:45.338253 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.338238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:01:45.338353 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.338331 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token" (OuterVolumeSpecName: "istio-token") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:45.338420 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.338406 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1db2e041-9041-4ff6-a026-b0274958d058" (UID: "1db2e041-9041-4ff6-a026-b0274958d058"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:01:45.437671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.437635 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.437671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.437663 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k8qq\" (UniqueName: \"kubernetes.io/projected/1db2e041-9041-4ff6-a026-b0274958d058-kube-api-access-8k8qq\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.437671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.437673 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.437671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.437682 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db2e041-9041-4ff6-a026-b0274958d058-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.437933 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.437690 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db2e041-9041-4ff6-a026-b0274958d058-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:01:45.557761 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.557666 2569 generic.go:356] "Generic (PLEG): container finished" podID="1db2e041-9041-4ff6-a026-b0274958d058" containerID="2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0" exitCode=0 Jul 07 18:01:45.557761 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.557730 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" Jul 07 18:01:45.557761 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.557748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerDied","Data":"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0"} Jul 07 18:01:45.557965 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.557829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5" event={"ID":"1db2e041-9041-4ff6-a026-b0274958d058","Type":"ContainerDied","Data":"528d8494cd5a5b277acf6c769e86ce2e65fd0226768a5cc3038c32fcdac8f844"} Jul 07 18:01:45.557965 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.557852 2569 scope.go:117] "RemoveContainer" containerID="2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0" Jul 07 18:01:45.566865 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.566847 2569 scope.go:117] "RemoveContainer" containerID="08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875" Jul 07 18:01:45.575213 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.575195 2569 scope.go:117] "RemoveContainer" containerID="2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0" Jul 07 18:01:45.575500 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:45.575477 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0\": container with ID starting with 2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0 not found: ID does not exist" containerID="2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0" Jul 07 18:01:45.575591 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.575507 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0"} err="failed to get container status \"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0\": rpc error: code = NotFound desc = could not find container \"2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0\": container with ID starting with 2d44a5bdf930135617ff8b5009e833f43bdb11e8fd5d77ed9d4ca7552ec5c3f0 not found: ID does not exist" Jul 07 18:01:45.575591 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.575526 2569 scope.go:117] "RemoveContainer" containerID="08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875" Jul 07 18:01:45.575785 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:01:45.575769 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875\": container with ID starting with 08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875 not found: ID does not exist" containerID="08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875" Jul 07 18:01:45.575823 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.575789 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875"} err="failed to get container status \"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875\": rpc error: code = NotFound desc = could not find container \"08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875\": container with ID starting with 08a3f1cff217136910e940c1a1a508bdcc8ad66ae492f841fc971c89591dc875 not found: ID does not exist" Jul 07 18:01:45.580156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.580136 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:45.584308 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:45.584274 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-97c1e9-openshift-default-6d6c687b69-qs7k5"] Jul 07 18:01:47.451562 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:47.451528 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db2e041-9041-4ff6-a026-b0274958d058" path="/var/lib/kubelet/pods/1db2e041-9041-4ff6-a026-b0274958d058/volumes" Jul 07 18:01:48.176643 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:48.176606 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:48.564861 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:48.564764 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:48.871154 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:48.871060 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:01:49.059155 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059114 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:01:49.059514 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059500 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:49.059585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059515 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86da86a5-2b5d-41b7-a5a9-74cf378d4e66" containerName="istio-proxy" Jul 07 18:01:49.059585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059525 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.059585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059530 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.059585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059549 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.059585 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059554 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.059846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059615 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.059846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.059624 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db2e041-9041-4ff6-a026-b0274958d058" containerName="istio-proxy" Jul 07 18:01:49.062951 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.062934 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.070963 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.070946 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-df746e-openshift-default-dockercfg-9tszk\"" Jul 07 18:01:49.096199 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.096157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:01:49.174758 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.174943 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.174943 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.174943 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.174943 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gznst\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.175100 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.175100 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.174983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.175100 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.175001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.175100 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.175033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.254233 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.254191 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:49.276411 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276411 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gznst\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.276669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.277020 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276908 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.277020 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276909 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.277020 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.276969 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.277210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.277191 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.277270 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.277233 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.278982 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.278963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.279209 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.279190 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.291094 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.291058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gznst\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.292890 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.292868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token\") pod \"e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.373690 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.373657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:49.376702 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.376671 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:49.549259 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.549234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:01:49.551504 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:01:49.551478 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c4cb1a_d17b_4164_8f85_60c7724b5c26.slice/crio-ce37ae6c2ce5f2fd22e52e24a13a71e46f2321d80e7b0e79a06e8f1a62d68faf WatchSource:0}: Error finding container ce37ae6c2ce5f2fd22e52e24a13a71e46f2321d80e7b0e79a06e8f1a62d68faf: Status 404 returned error can't find the container with id ce37ae6c2ce5f2fd22e52e24a13a71e46f2321d80e7b0e79a06e8f1a62d68faf Jul 07 18:01:49.553662 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.553631 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:49.553720 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.553695 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:49.553763 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.553726 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:49.577920 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:49.577895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerStarted","Data":"ce37ae6c2ce5f2fd22e52e24a13a71e46f2321d80e7b0e79a06e8f1a62d68faf"} Jul 07 18:01:50.584041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:50.584004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerStarted","Data":"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a"} Jul 07 18:01:50.625597 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:50.625538 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podStartSLOduration=1.625520791 podStartE2EDuration="1.625520791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:50.623183671 +0000 UTC m=+2375.743388015" watchObservedRunningTime="2026-07-07 18:01:50.625520791 +0000 UTC m=+2375.745725101" Jul 07 18:01:51.374622 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:51.374584 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:51.380136 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:51.380109 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:51.588923 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:51.588882 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:51.589929 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:51.589905 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:01:52.835911 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:52.835869 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:52.936126 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:52.936092 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:53.020465 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:53.020426 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:53.172810 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:53.172776 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:53.705933 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:53.705880 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:53.807918 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:53.807881 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:57.297948 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.297828 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:57.542297 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.542256 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 18:01:57.542476 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.542344 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 18:01:57.628654 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.628560 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerID="5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd" exitCode=0 Jul 07 18:01:57.628654 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.628628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerDied","Data":"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd"} Jul 07 18:01:57.629068 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.629054 2569 scope.go:117] "RemoveContainer" containerID="5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd" Jul 07 18:01:57.630199 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.630170 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:57.630266 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.630242 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:57.630333 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:57.630317 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:01:58.635346 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:58.635308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerStarted","Data":"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb"} Jul 07 18:01:59.182302 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:59.182253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:01:59.254375 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:59.254331 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:01:59.254624 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:59.254598 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 18:01:59.254675 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:01:59.254655 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 18:02:00.254079 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:00.254033 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 18:02:00.254497 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:00.254098 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 18:02:01.254797 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:01.254756 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 18:02:01.255205 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:01.254815 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 18:02:02.254566 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:02.254525 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 07 18:02:02.254751 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:02.254584 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 07 18:02:03.257937 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:03.257902 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:02:03.258440 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:03.258307 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:02:03.259043 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:03.259024 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:02:05.710392 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:05.710344 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:05.814828 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:05.814795 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:07.087963 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.087710 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:38744->10.132.0.55:15021: read: connection reset by peer" start-of-body= Jul 07 18:02:07.087963 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.087778 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:38744->10.132.0.55:15021: read: connection reset by peer" Jul 07 18:02:07.682792 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.682760 2569 generic.go:356] "Generic (PLEG): container finished" podID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerID="50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a" exitCode=0 Jul 07 18:02:07.682792 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.682805 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerDied","Data":"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a"} Jul 07 18:02:07.683179 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.683165 2569 scope.go:117] "RemoveContainer" containerID="50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a" Jul 07 18:02:07.684218 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.684190 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:02:07.684268 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.684260 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:02:07.684331 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:07.684316 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 18:02:08.689153 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:08.689121 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerStarted","Data":"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15"} Jul 07 18:02:09.374018 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:09.373981 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:02:09.374320 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:09.374269 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 07 18:02:09.374379 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:09.374347 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 07 18:02:10.374858 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:10.374819 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 07 18:02:10.375245 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:10.374881 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 07 18:02:11.375053 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.375012 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 07 18:02:11.375466 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.375089 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 07 18:02:11.773266 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.773229 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:02:11.775841 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.775821 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.779360 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.779338 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"default-dockercfg-xjxs7\"" Jul 07 18:02:11.779488 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.779453 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"openshift-service-ca.crt\"" Jul 07 18:02:11.780352 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.780336 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"kube-root-ca.crt\"" Jul 07 18:02:11.782889 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.782869 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"auth-test-model-1e464b-kserve-self-signed-certs\"" Jul 07 18:02:11.805311 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.797650 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:02:11.807802 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.807769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:11.924072 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.924275 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.924275 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.924275 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m277l\" (UniqueName: \"kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.924275 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:11.924275 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:11.924240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.025804 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.025804 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.025804 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m277l\" (UniqueName: \"kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026087 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026087 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025859 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026087 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.025946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026245 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.026176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026339 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.026242 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.026339 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.026276 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.028247 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.028222 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.028584 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.028565 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.035701 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.035683 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m277l\" (UniqueName: \"kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l\") pod \"auth-test-model-1e464b-kserve-67df599d75-tgxjg\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.085802 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.085763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:12.270505 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.270481 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:02:12.272635 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:02:12.272608 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb42db08_011c_4aae_9ae5_bd943f361c1e.slice/crio-d17fab8155a42ea18ae6823fec21961014df0110417b389533c128e53d3311e7 WatchSource:0}: Error finding container d17fab8155a42ea18ae6823fec21961014df0110417b389533c128e53d3311e7: Status 404 returned error can't find the container with id d17fab8155a42ea18ae6823fec21961014df0110417b389533c128e53d3311e7 Jul 07 18:02:12.375020 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.374979 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 07 18:02:12.375156 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.375058 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 07 18:02:12.716776 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.716741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" event={"ID":"bb42db08-011c-4aae-9ae5-bd943f361c1e","Type":"ContainerStarted","Data":"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9"} Jul 07 18:02:12.716776 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.716783 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" event={"ID":"bb42db08-011c-4aae-9ae5-bd943f361c1e","Type":"ContainerStarted","Data":"d17fab8155a42ea18ae6823fec21961014df0110417b389533c128e53d3311e7"} Jul 07 18:02:12.738941 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:12.738883 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" podStartSLOduration=1.738868497 podStartE2EDuration="1.738868497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:02:12.736822097 +0000 UTC m=+2397.857026427" watchObservedRunningTime="2026-07-07 18:02:12.738868497 +0000 UTC m=+2397.859072827" Jul 07 18:02:13.378730 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:13.378684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:02:13.379175 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:13.379122 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:02:13.379869 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:13.379847 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:02:14.748382 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:14.748347 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:14.848055 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:14.848019 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:15.290528 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:15.290491 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:02:15.580422 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:15.580332 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 18:02:15.586977 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:15.586949 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 18:02:22.086228 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:22.086195 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:22.086680 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:22.086246 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:22.098950 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:22.098915 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:22.777610 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:22.777571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:02:48.293428 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:48.293395 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:51.834652 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:02:51.834617 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:03:13.195801 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:03:13.195763 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:03:29.090397 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:03:29.090366 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:03:46.486651 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:03:46.486614 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:04:03.488990 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:03.488945 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:04:21.393602 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:21.393502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:04:39.600640 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:39.600594 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:04:53.845299 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:53.845240 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:55.696432 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:55.696396 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:04:55.942137 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:55.942104 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:59.199732 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:59.199699 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:04:59.227951 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:04:59.227861 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:00.234157 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:00.234115 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:12.690677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:12.690641 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:05:14.221257 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.221186 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" podUID="bb42db08-011c-4aae-9ae5-bd943f361c1e" containerName="main" containerID="cri-o://ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9" gracePeriod=45 Jul 07 18:05:14.236755 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.236728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:14.481679 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.481615 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:05:14.572793 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.572793 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.573075 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572825 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.573075 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572898 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m277l\" (UniqueName: \"kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.573075 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572945 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.573075 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.572974 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm\") pod \"bb42db08-011c-4aae-9ae5-bd943f361c1e\" (UID: \"bb42db08-011c-4aae-9ae5-bd943f361c1e\") " Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573079 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573090 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home" (OuterVolumeSpecName: "home") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573202 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573276 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-tmp-dir\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573318 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-home\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:14.573340 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.573327 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-model-cache\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:14.575412 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.575385 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm" (OuterVolumeSpecName: "dshm") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:14.575412 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.575401 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:05:14.575556 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.575444 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l" (OuterVolumeSpecName: "kube-api-access-m277l") pod "bb42db08-011c-4aae-9ae5-bd943f361c1e" (UID: "bb42db08-011c-4aae-9ae5-bd943f361c1e"). InnerVolumeSpecName "kube-api-access-m277l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:05:14.610354 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.610317 2569 generic.go:356] "Generic (PLEG): container finished" podID="bb42db08-011c-4aae-9ae5-bd943f361c1e" containerID="ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9" exitCode=0 Jul 07 18:05:14.610354 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.610354 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" Jul 07 18:05:14.610542 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.610390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" event={"ID":"bb42db08-011c-4aae-9ae5-bd943f361c1e","Type":"ContainerDied","Data":"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9"} Jul 07 18:05:14.610542 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.610436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg" event={"ID":"bb42db08-011c-4aae-9ae5-bd943f361c1e","Type":"ContainerDied","Data":"d17fab8155a42ea18ae6823fec21961014df0110417b389533c128e53d3311e7"} Jul 07 18:05:14.610542 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.610453 2569 scope.go:117] "RemoveContainer" containerID="ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9" Jul 07 18:05:14.620131 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.620110 2569 scope.go:117] "RemoveContainer" containerID="ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9" Jul 07 18:05:14.620463 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:05:14.620441 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9\": container with ID starting with ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9 not found: ID does not exist" containerID="ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9" Jul 07 18:05:14.620527 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.620473 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9"} err="failed to get container status \"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9\": rpc error: code = NotFound desc = could not find container \"ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9\": container with ID starting with ddddd76709aaa837cc0e188c048f678a91927fe15ac76b2610d031e5385b92c9 not found: ID does not exist" Jul 07 18:05:14.637501 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.637473 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:05:14.639188 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.639167 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/auth-test-model-1e464b-kserve-67df599d75-tgxjg"] Jul 07 18:05:14.674249 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.674221 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb42db08-011c-4aae-9ae5-bd943f361c1e-tls-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:14.674249 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.674248 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m277l\" (UniqueName: \"kubernetes.io/projected/bb42db08-011c-4aae-9ae5-bd943f361c1e-kube-api-access-m277l\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:14.674461 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:14.674258 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb42db08-011c-4aae-9ae5-bd943f361c1e-dshm\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.452543 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:15.452512 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb42db08-011c-4aae-9ae5-bd943f361c1e" path="/var/lib/kubelet/pods/bb42db08-011c-4aae-9ae5-bd943f361c1e/volumes" Jul 07 18:05:25.097066 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:25.097034 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:05:28.876219 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.876185 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:05:28.876621 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.876608 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb42db08-011c-4aae-9ae5-bd943f361c1e" containerName="main" Jul 07 18:05:28.876621 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.876622 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb42db08-011c-4aae-9ae5-bd943f361c1e" containerName="main" Jul 07 18:05:28.876722 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.876711 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb42db08-011c-4aae-9ae5-bd943f361c1e" containerName="main" Jul 07 18:05:28.878697 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.878680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:28.881333 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.881308 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"sub-test-model-df746e-kserve-self-signed-certs\"" Jul 07 18:05:28.882535 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.882514 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"kube-root-ca.crt\"" Jul 07 18:05:28.882792 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.882771 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"openshift-service-ca.crt\"" Jul 07 18:05:28.882885 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.882832 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"default-dockercfg-k2r8l\"" Jul 07 18:05:28.891351 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.891328 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:05:28.923826 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:28.923786 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:29.005248 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.005449 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.005449 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.005449 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.005449 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005360 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5r5\" (UniqueName: \"kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.005589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.005462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106351 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106536 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106536 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106536 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106671 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5r5\" (UniqueName: \"kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106981 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106949 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.106981 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.107390 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.106990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.108862 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.108831 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.109227 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.109206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.117618 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.117599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5r5\" (UniqueName: \"kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5\") pod \"sub-test-model-df746e-kserve-65cdd47f-9gm9p\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.191359 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.191323 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:29.344079 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.344047 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:05:29.686117 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.686082 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" event={"ID":"4539280d-6d56-49e5-9c4f-2fe763dc92df","Type":"ContainerStarted","Data":"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760"} Jul 07 18:05:29.686117 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.686126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" event={"ID":"4539280d-6d56-49e5-9c4f-2fe763dc92df","Type":"ContainerStarted","Data":"ffb4269815d229b75dead7810e1c96b104fb61a14ca7d043753d2ff2c8168cba"} Jul 07 18:05:29.710514 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:29.710463 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" podStartSLOduration=1.71044701 podStartE2EDuration="1.71044701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:05:29.708434213 +0000 UTC m=+2594.828638544" watchObservedRunningTime="2026-07-07 18:05:29.71044701 +0000 UTC m=+2594.830651339" Jul 07 18:05:34.543843 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:34.543805 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:36.043730 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:36.043644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:05:39.191915 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:39.191841 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:39.192400 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:39.191937 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:39.205237 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:39.205210 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:39.747170 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:39.747137 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:05:43.953147 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:43.953089 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:44.047392 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:44.047357 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:55.388573 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:05:55.388536 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:06:13.985246 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:13.985205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:06:14.696081 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:14.696045 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:20.555742 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:20.555707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:21.500872 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:21.500832 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:06:21.575338 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:21.575304 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:25.125816 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:25.125779 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:32.301968 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:32.301931 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:06:36.521823 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.521773 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" podUID="4539280d-6d56-49e5-9c4f-2fe763dc92df" containerName="main" containerID="cri-o://bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760" gracePeriod=45 Jul 07 18:06:36.778956 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.778884 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:06:36.855594 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855556 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.855594 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.855837 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.855837 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.855837 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855749 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.855837 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855773 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td5r5\" (UniqueName: \"kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5\") pod \"4539280d-6d56-49e5-9c4f-2fe763dc92df\" (UID: \"4539280d-6d56-49e5-9c4f-2fe763dc92df\") " Jul 07 18:06:36.856055 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855925 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache" (OuterVolumeSpecName: "model-cache") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:36.856055 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.855940 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home" (OuterVolumeSpecName: "home") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:36.856161 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.856082 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:36.856161 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.856111 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-model-cache\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.856161 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.856133 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-home\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.858029 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.858007 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm" (OuterVolumeSpecName: "dshm") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:36.858109 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.858030 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:06:36.858149 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.858107 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5" (OuterVolumeSpecName: "kube-api-access-td5r5") pod "4539280d-6d56-49e5-9c4f-2fe763dc92df" (UID: "4539280d-6d56-49e5-9c4f-2fe763dc92df"). InnerVolumeSpecName "kube-api-access-td5r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:06:36.956629 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.956589 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-dshm\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.956629 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.956621 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4539280d-6d56-49e5-9c4f-2fe763dc92df-tmp-dir\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.956629 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.956632 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5r5\" (UniqueName: \"kubernetes.io/projected/4539280d-6d56-49e5-9c4f-2fe763dc92df-kube-api-access-td5r5\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.956629 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:36.956641 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4539280d-6d56-49e5-9c4f-2fe763dc92df-tls-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:06:37.019503 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.019467 2569 generic.go:356] "Generic (PLEG): container finished" podID="4539280d-6d56-49e5-9c4f-2fe763dc92df" containerID="bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760" exitCode=0 Jul 07 18:06:37.019727 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.019508 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" Jul 07 18:06:37.019727 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.019552 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" event={"ID":"4539280d-6d56-49e5-9c4f-2fe763dc92df","Type":"ContainerDied","Data":"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760"} Jul 07 18:06:37.019727 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.019611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p" event={"ID":"4539280d-6d56-49e5-9c4f-2fe763dc92df","Type":"ContainerDied","Data":"ffb4269815d229b75dead7810e1c96b104fb61a14ca7d043753d2ff2c8168cba"} Jul 07 18:06:37.019727 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.019635 2569 scope.go:117] "RemoveContainer" containerID="bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760" Jul 07 18:06:37.030210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.030190 2569 scope.go:117] "RemoveContainer" containerID="bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760" Jul 07 18:06:37.030533 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:06:37.030512 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760\": container with ID starting with bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760 not found: ID does not exist" containerID="bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760" Jul 07 18:06:37.030614 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.030542 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760"} err="failed to get container status \"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760\": rpc error: code = NotFound desc = could not find container \"bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760\": container with ID starting with bde73ddec0b97e928d8a57f623a2bfad42791ff3012f915d2f4ac941dffa8760 not found: ID does not exist" Jul 07 18:06:37.053432 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.053396 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:06:37.060548 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.060516 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/sub-test-model-df746e-kserve-65cdd47f-9gm9p"] Jul 07 18:06:37.451500 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:37.451467 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4539280d-6d56-49e5-9c4f-2fe763dc92df" path="/var/lib/kubelet/pods/4539280d-6d56-49e5-9c4f-2fe763dc92df/volumes" Jul 07 18:06:50.817432 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.817397 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:06:50.817899 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.817804 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4539280d-6d56-49e5-9c4f-2fe763dc92df" containerName="main" Jul 07 18:06:50.817899 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.817816 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4539280d-6d56-49e5-9c4f-2fe763dc92df" containerName="main" Jul 07 18:06:50.817899 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.817881 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4539280d-6d56-49e5-9c4f-2fe763dc92df" containerName="main" Jul 07 18:06:50.821392 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.821373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.824979 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.824944 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"openshift-service-ca.crt\"" Jul 07 18:06:50.825132 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.825003 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"default-dockercfg-k2r8l\"" Jul 07 18:06:50.826360 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.826343 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"rate-test-model-df746e-kserve-self-signed-certs\"" Jul 07 18:06:50.826448 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.826346 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"kube-root-ca.crt\"" Jul 07 18:06:50.839370 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.839344 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:06:50.887751 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.887945 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887766 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.887945 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdkn\" (UniqueName: \"kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.887945 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.887945 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.888107 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.887989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989092 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989092 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989403 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdkn\" (UniqueName: \"kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989403 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989403 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989403 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989637 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989694 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.989744 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.989714 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.991616 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.991575 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:50.991922 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.991897 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:50.992033 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:50.991964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:51.000387 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:51.000362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdkn\" (UniqueName: \"kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn\") pod \"rate-test-model-df746e-kserve-857d9497d7-g9vgf\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:51.131005 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:51.130913 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:06:51.283900 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:51.283865 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:06:51.286727 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:06:51.286695 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9337c697_58f5_4e28_911b_cb0bff0eb466.slice/crio-a21752c707d9ac0f53bf8dd677ccc8c383f59f7d14c8db3944b4841f757de034 WatchSource:0}: Error finding container a21752c707d9ac0f53bf8dd677ccc8c383f59f7d14c8db3944b4841f757de034: Status 404 returned error can't find the container with id a21752c707d9ac0f53bf8dd677ccc8c383f59f7d14c8db3944b4841f757de034 Jul 07 18:06:52.098526 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:52.098487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" event={"ID":"9337c697-58f5-4e28-911b-cb0bff0eb466","Type":"ContainerStarted","Data":"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e"} Jul 07 18:06:52.098526 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:52.098533 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" event={"ID":"9337c697-58f5-4e28-911b-cb0bff0eb466","Type":"ContainerStarted","Data":"a21752c707d9ac0f53bf8dd677ccc8c383f59f7d14c8db3944b4841f757de034"} Jul 07 18:06:52.133370 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:52.133321 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" podStartSLOduration=2.133305421 podStartE2EDuration="2.133305421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:06:52.129768095 +0000 UTC m=+2677.249972416" watchObservedRunningTime="2026-07-07 18:06:52.133305421 +0000 UTC m=+2677.253509741" Jul 07 18:06:55.090848 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:06:55.090810 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:07:01.131854 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:01.131809 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:07:01.132262 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:01.131869 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:07:01.144511 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:01.144486 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:07:01.155688 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:01.155658 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:07:06.628011 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:06.627975 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:06.734005 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:06.733961 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:15.423587 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:15.423540 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:15.610457 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:15.610422 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 18:07:15.617579 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:15.617553 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 18:07:15.823720 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:15.823667 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:15.926686 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:15.926611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:26.214512 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:26.214475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:07:38.703210 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:38.703172 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:07:46.120450 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:46.120403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:47.049941 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:47.049906 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:07:47.074483 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:47.074452 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:48.275168 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:48.275128 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:48.679916 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:48.679886 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:48.784041 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:48.784000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:48.884015 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:48.883978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:49.061965 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:49.061877 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:53.092942 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:53.092909 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:07:53.868122 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:07:53.868078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:02.070321 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.070232 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" podUID="9337c697-58f5-4e28-911b-cb0bff0eb466" containerName="main" containerID="cri-o://6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e" gracePeriod=45 Jul 07 18:08:02.322662 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.322601 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:08:02.367304 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367257 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfdkn\" (UniqueName: \"kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367491 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367337 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367491 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367368 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367491 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367392 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367491 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367449 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367748 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367493 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir\") pod \"9337c697-58f5-4e28-911b-cb0bff0eb466\" (UID: \"9337c697-58f5-4e28-911b-cb0bff0eb466\") " Jul 07 18:08:02.367748 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367562 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home" (OuterVolumeSpecName: "home") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:02.367858 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367792 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache" (OuterVolumeSpecName: "model-cache") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:02.367858 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367843 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-home\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.367969 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.367899 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:02.369625 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.369600 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm" (OuterVolumeSpecName: "dshm") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:02.369753 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.369624 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn" (OuterVolumeSpecName: "kube-api-access-qfdkn") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "kube-api-access-qfdkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:02.369753 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.369691 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9337c697-58f5-4e28-911b-cb0bff0eb466" (UID: "9337c697-58f5-4e28-911b-cb0bff0eb466"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:08:02.450092 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.450048 2569 generic.go:356] "Generic (PLEG): container finished" podID="9337c697-58f5-4e28-911b-cb0bff0eb466" containerID="6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e" exitCode=0 Jul 07 18:08:02.450092 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.450089 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" Jul 07 18:08:02.450363 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.450140 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" event={"ID":"9337c697-58f5-4e28-911b-cb0bff0eb466","Type":"ContainerDied","Data":"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e"} Jul 07 18:08:02.450363 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.450186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf" event={"ID":"9337c697-58f5-4e28-911b-cb0bff0eb466","Type":"ContainerDied","Data":"a21752c707d9ac0f53bf8dd677ccc8c383f59f7d14c8db3944b4841f757de034"} Jul 07 18:08:02.450363 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.450205 2569 scope.go:117] "RemoveContainer" containerID="6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e" Jul 07 18:08:02.461960 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.461939 2569 scope.go:117] "RemoveContainer" containerID="6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e" Jul 07 18:08:02.462379 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:08:02.462356 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e\": container with ID starting with 6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e not found: ID does not exist" containerID="6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e" Jul 07 18:08:02.462442 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.462388 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e"} err="failed to get container status \"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e\": rpc error: code = NotFound desc = could not find container \"6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e\": container with ID starting with 6dec9b53f9db06cd8ea04b0e8426ab631e098053387c8230930051d4f8cc092e not found: ID does not exist" Jul 07 18:08:02.468535 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.468506 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-dshm\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.468535 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.468539 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337c697-58f5-4e28-911b-cb0bff0eb466-tls-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.468635 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.468550 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-model-cache\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.468635 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.468558 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337c697-58f5-4e28-911b-cb0bff0eb466-tmp-dir\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.468635 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.468566 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdkn\" (UniqueName: \"kubernetes.io/projected/9337c697-58f5-4e28-911b-cb0bff0eb466-kube-api-access-qfdkn\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:02.479514 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.479492 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:08:02.485681 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:02.485662 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/rate-test-model-df746e-kserve-857d9497d7-g9vgf"] Jul 07 18:08:03.452033 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:03.452001 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9337c697-58f5-4e28-911b-cb0bff0eb466" path="/var/lib/kubelet/pods/9337c697-58f5-4e28-911b-cb0bff0eb466/volumes" Jul 07 18:08:31.377685 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:31.377639 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:31.665120 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:31.665083 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:08:31.665399 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:31.665375 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" containerID="cri-o://1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb" gracePeriod=30 Jul 07 18:08:31.670159 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:31.670117 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:31.923884 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:31.923781 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:32.121144 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:32.121107 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:32.554209 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:32.554175 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:32.721145 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:32.721110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:32.826997 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:32.826918 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:33.079151 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:33.079053 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:33.260187 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:33.260144 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:08:33.260376 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:33.260211 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:08:35.700709 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:35.700671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8ftkv"] Jul 07 18:08:36.923979 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:36.923954 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:08:37.093663 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093581 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093663 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093627 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093663 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093658 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093716 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7csr\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093733 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093813 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093866 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.093931 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093905 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token\") pod \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\" (UID: \"e5c2caf4-c8bb-4ea6-8234-89badf791bc5\") " Jul 07 18:08:37.094211 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093938 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:37.094211 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.093999 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:37.096641 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.094715 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:08:37.096641 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.094787 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:37.096641 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.094888 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.096641 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.094918 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.096641 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.095672 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:37.096968 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.096816 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr" (OuterVolumeSpecName: "kube-api-access-z7csr") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "kube-api-access-z7csr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:37.097426 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.097387 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:37.102347 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.100774 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:37.103934 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.103014 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5c2caf4-c8bb-4ea6-8234-89badf791bc5" (UID: "e5c2caf4-c8bb-4ea6-8234-89badf791bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:08:37.195846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195804 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.195846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195839 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7csr\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-kube-api-access-z7csr\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.195846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195850 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.196095 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195859 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.196095 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195867 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.196095 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195876 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.196095 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.195884 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2caf4-c8bb-4ea6-8234-89badf791bc5-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:37.617336 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.617301 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerID="1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb" exitCode=0 Jul 07 18:08:37.617510 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.617366 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" Jul 07 18:08:37.617510 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.617380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerDied","Data":"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb"} Jul 07 18:08:37.617510 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.617436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl" event={"ID":"e5c2caf4-c8bb-4ea6-8234-89badf791bc5","Type":"ContainerDied","Data":"e93d123fb8b699398365d6ee21b308ee6a6c0a3a349cc25d69ec2c6659d4325e"} Jul 07 18:08:37.617510 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.617459 2569 scope.go:117] "RemoveContainer" containerID="1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb" Jul 07 18:08:37.628945 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.628922 2569 scope.go:117] "RemoveContainer" containerID="5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd" Jul 07 18:08:37.637518 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.637499 2569 scope.go:117] "RemoveContainer" containerID="1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb" Jul 07 18:08:37.637782 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:08:37.637763 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb\": container with ID starting with 1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb not found: ID does not exist" containerID="1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb" Jul 07 18:08:37.637846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.637787 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb"} err="failed to get container status \"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb\": rpc error: code = NotFound desc = could not find container \"1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb\": container with ID starting with 1547c947c1432f05a5a08ffdd177ec6d81b23b2e44e5988ad7a80520e24827fb not found: ID does not exist" Jul 07 18:08:37.637846 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.637803 2569 scope.go:117] "RemoveContainer" containerID="5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd" Jul 07 18:08:37.638012 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:08:37.637995 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd\": container with ID starting with 5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd not found: ID does not exist" containerID="5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd" Jul 07 18:08:37.638055 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.638020 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd"} err="failed to get container status \"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd\": rpc error: code = NotFound desc = could not find container \"5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd\": container with ID starting with 5ee37b8d8d24e566e0f139b01dbc7b39bacb18c7425f7b9c516c79db787d20dd not found: ID does not exist" Jul 07 18:08:37.641669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.641648 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:08:37.650052 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:37.650032 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1e464b-openshift-default-78b466f5-76wzl"] Jul 07 18:08:39.451297 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:39.451254 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" path="/var/lib/kubelet/pods/e5c2caf4-c8bb-4ea6-8234-89badf791bc5/volumes" Jul 07 18:08:40.371013 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.370974 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:40.450300 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.450252 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:40.636690 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.636663 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:08:40.637150 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.636918 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" containerID="cri-o://dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15" gracePeriod=30 Jul 07 18:08:40.666302 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.666254 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:40.892751 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:40.892634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:43.381413 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:43.381372 2569 patch_prober.go:28] interesting pod/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:08:43.381780 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:43.381427 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:08:45.893888 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:45.893865 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:08:46.076068 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.075969 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076068 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076041 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076074 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076092 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076125 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076150 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076185 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gznst\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076267 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert\") pod \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\" (UID: \"89c4cb1a-d17b-4164-8f85-60c7724b5c26\") " Jul 07 18:08:46.076677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076318 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:46.076677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076492 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:46.076677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076592 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:46.076677 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076667 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-credential-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.076877 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076690 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-certs\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.076877 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.076800 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:08:46.077489 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.077461 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data" (OuterVolumeSpecName: "istio-data") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:46.078485 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.078453 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:08:46.078627 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.078605 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token" (OuterVolumeSpecName: "istio-token") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:46.078908 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.078889 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:46.078961 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.078903 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst" (OuterVolumeSpecName: "kube-api-access-gznst") pod "89c4cb1a-d17b-4164-8f85-60c7724b5c26" (UID: "89c4cb1a-d17b-4164-8f85-60c7724b5c26"). InnerVolumeSpecName "kube-api-access-gznst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:46.177687 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177651 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istiod-ca-cert\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177687 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177678 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-podinfo\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177687 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177690 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-envoy\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177921 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177702 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-token\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177921 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177713 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-istio-data\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177921 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177724 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gznst\" (UniqueName: \"kubernetes.io/projected/89c4cb1a-d17b-4164-8f85-60c7724b5c26-kube-api-access-gznst\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.177921 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.177735 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c4cb1a-d17b-4164-8f85-60c7724b5c26-workload-socket\") on node \"ip-10-0-137-197.ec2.internal\" DevicePath \"\"" Jul 07 18:08:46.665896 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.665857 2569 generic.go:356] "Generic (PLEG): container finished" podID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerID="dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15" exitCode=0 Jul 07 18:08:46.666073 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.665976 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerDied","Data":"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15"} Jul 07 18:08:46.666073 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.665982 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" Jul 07 18:08:46.666073 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.666023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4" event={"ID":"89c4cb1a-d17b-4164-8f85-60c7724b5c26","Type":"ContainerDied","Data":"ce37ae6c2ce5f2fd22e52e24a13a71e46f2321d80e7b0e79a06e8f1a62d68faf"} Jul 07 18:08:46.666073 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.666046 2569 scope.go:117] "RemoveContainer" containerID="dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15" Jul 07 18:08:46.676522 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.676503 2569 scope.go:117] "RemoveContainer" containerID="50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a" Jul 07 18:08:46.685387 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.685371 2569 scope.go:117] "RemoveContainer" containerID="dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15" Jul 07 18:08:46.685639 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:08:46.685618 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15\": container with ID starting with dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15 not found: ID does not exist" containerID="dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15" Jul 07 18:08:46.685683 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.685646 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15"} err="failed to get container status \"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15\": rpc error: code = NotFound desc = could not find container \"dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15\": container with ID starting with dddddc2d2debec6cec8397cdb8df681891de8c4495d58da03247f79c7078cc15 not found: ID does not exist" Jul 07 18:08:46.685683 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.685662 2569 scope.go:117] "RemoveContainer" containerID="50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a" Jul 07 18:08:46.685869 ip-10-0-137-197 kubenswrapper[2569]: E0707 18:08:46.685854 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a\": container with ID starting with 50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a not found: ID does not exist" containerID="50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a" Jul 07 18:08:46.685905 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.685873 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a"} err="failed to get container status \"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a\": rpc error: code = NotFound desc = could not find container \"50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a\": container with ID starting with 50adafa1fbdada4a38eff277262d5c274d367c112995c0eaff518ec322bb431a not found: ID does not exist" Jul 07 18:08:46.694824 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.694803 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:08:46.696509 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:46.696489 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-df746e-openshift-default-7b7bcb54c9-jqjc4"] Jul 07 18:08:47.451519 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:47.451489 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" path="/var/lib/kubelet/pods/89c4cb1a-d17b-4164-8f85-60c7724b5c26/volumes" Jul 07 18:08:47.501420 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:47.501395 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-k8qxq_402b619f-1f41-4c4b-935d-522b81b6e4b5/copy-manifests/0.log" Jul 07 18:08:47.509547 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:47.509524 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-k8qxq_402b619f-1f41-4c4b-935d-522b81b6e4b5/manager/0.log" Jul 07 18:08:49.303937 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:49.303905 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bszrv_8e9f77d0-856e-46d5-8916-d6d045e2cd4c/manager/0.log" Jul 07 18:08:49.824798 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:49.824766 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8ftkv_91705533-682f-4b23-9d5e-b7764c12a94a/limitador/0.log" Jul 07 18:08:50.358486 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:50.358458 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rlb6w_4f7a0f87-b8ec-405b-9e62-c84aed63a0cf/istio-proxy/0.log" Jul 07 18:08:51.313154 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:51.313125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-784748979c-znp9d_bb2371ec-c2d8-4094-a9e4-939d0601b6d5/router/0.log" Jul 07 18:08:51.891459 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:51.891427 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xw9sr_9f72135d-0228-48ad-8c89-240284ab53b9/main/0.log" Jul 07 18:08:52.145939 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:52.145820 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-s7xrq_4fced498-f4db-46dd-a055-26fdab2b9d37/main/0.log" Jul 07 18:08:52.266069 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:52.266037 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-q4jtt_f386777a-7c44-4e75-8c21-cdd70d7ec66e/main/0.log" Jul 07 18:08:56.551735 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.551695 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qlcl8/must-gather-qs2bn"] Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552059 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9337c697-58f5-4e28-911b-cb0bff0eb466" containerName="main" Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552071 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9337c697-58f5-4e28-911b-cb0bff0eb466" containerName="main" Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552082 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552087 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552094 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.552130 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552100 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552165 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552174 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552179 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9337c697-58f5-4e28-911b-cb0bff0eb466" containerName="main" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552186 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552191 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552251 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552256 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c2caf4-c8bb-4ea6-8234-89badf791bc5" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552264 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.552369 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.552269 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c4cb1a-d17b-4164-8f85-60c7724b5c26" containerName="istio-proxy" Jul 07 18:08:56.555357 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.555339 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.563021 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.563000 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qlcl8\"/\"default-dockercfg-xx92f\"" Jul 07 18:08:56.563145 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.563026 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qlcl8\"/\"openshift-service-ca.crt\"" Jul 07 18:08:56.563145 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.563046 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qlcl8\"/\"kube-root-ca.crt\"" Jul 07 18:08:56.569382 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.569354 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:56.588589 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.588560 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/must-gather-qs2bn"] Jul 07 18:08:56.656376 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.656332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:56.672596 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.672571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51c9fc07-21c8-4aaf-b25f-7959cc61535b-must-gather-output\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.672723 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.672603 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzsd\" (UniqueName: \"kubernetes.io/projected/51c9fc07-21c8-4aaf-b25f-7959cc61535b-kube-api-access-5tzsd\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.773539 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.773500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51c9fc07-21c8-4aaf-b25f-7959cc61535b-must-gather-output\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.773704 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.773547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzsd\" (UniqueName: \"kubernetes.io/projected/51c9fc07-21c8-4aaf-b25f-7959cc61535b-kube-api-access-5tzsd\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.773972 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.773952 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51c9fc07-21c8-4aaf-b25f-7959cc61535b-must-gather-output\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.783499 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.783472 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzsd\" (UniqueName: \"kubernetes.io/projected/51c9fc07-21c8-4aaf-b25f-7959cc61535b-kube-api-access-5tzsd\") pod \"must-gather-qs2bn\" (UID: \"51c9fc07-21c8-4aaf-b25f-7959cc61535b\") " pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:56.864902 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:56.864803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" Jul 07 18:08:57.014657 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:57.014631 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/must-gather-qs2bn"] Jul 07 18:08:57.016748 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:08:57.016725 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c9fc07_21c8_4aaf_b25f_7959cc61535b.slice/crio-f7a54d8032cc63eccd6586bc64def2fd982e816d63e82748d929f95a25595088 WatchSource:0}: Error finding container f7a54d8032cc63eccd6586bc64def2fd982e816d63e82748d929f95a25595088: Status 404 returned error can't find the container with id f7a54d8032cc63eccd6586bc64def2fd982e816d63e82748d929f95a25595088 Jul 07 18:08:57.019934 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:57.019913 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:08:57.718372 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:57.718334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" event={"ID":"51c9fc07-21c8-4aaf-b25f-7959cc61535b","Type":"ContainerStarted","Data":"f7a54d8032cc63eccd6586bc64def2fd982e816d63e82748d929f95a25595088"} Jul 07 18:08:58.727013 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:58.726976 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" event={"ID":"51c9fc07-21c8-4aaf-b25f-7959cc61535b","Type":"ContainerStarted","Data":"9245c2bdf69f9e109f30951b180a12b231180c969aabf7efdb4cf23b625ca9d6"} Jul 07 18:08:58.727431 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:58.727028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" event={"ID":"51c9fc07-21c8-4aaf-b25f-7959cc61535b","Type":"ContainerStarted","Data":"e952205f3db7030c67dfbadac6035226fd0add4453a40e8131b1703a0e902938"} Jul 07 18:08:58.752107 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:58.752055 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qlcl8/must-gather-qs2bn" podStartSLOduration=1.765255488 podStartE2EDuration="2.752038397s" totalImagesPullingTime="986.782909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:08:58.749733607 +0000 UTC m=+2803.869937938" watchObservedRunningTime="2026-07-07 18:08:58.752038397 +0000 UTC m=+2803.872242731" Jul 07 18:08:59.863689 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:59.863655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h8x4x_87d5a160-779e-4370-b6fb-96a892b2dbca/global-pull-secret-syncer/0.log" Jul 07 18:08:59.941359 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:08:59.941323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pbt7g_66ff5915-cd61-45ea-8199-1b41e636c35f/konnectivity-agent/0.log" Jul 07 18:09:00.121995 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:00.121919 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-197.ec2.internal_ad89d569afe56b30feba4f95595fae64/haproxy/0.log" Jul 07 18:09:05.559877 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:05.559842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bszrv_8e9f77d0-856e-46d5-8916-d6d045e2cd4c/manager/0.log" Jul 07 18:09:05.817798 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:05.817707 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8ftkv_91705533-682f-4b23-9d5e-b7764c12a94a/limitador/0.log" Jul 07 18:09:07.948027 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:07.947986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-mtbfz_a9cfcfec-4f6b-41e9-8ccb-4d9e93ebd6b4/cluster-monitoring-operator/0.log" Jul 07 18:09:08.188704 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:08.188659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/node-exporter/0.log" Jul 07 18:09:08.237963 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:08.237880 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/kube-rbac-proxy/0.log" Jul 07 18:09:08.276875 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:08.276841 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-42hn9_78d1c5cf-48f4-4b8a-ba0d-88629c0bbefd/init-textfile/0.log" Jul 07 18:09:09.159744 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.159706 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-f9x6t_01e94d75-ad29-4c3d-af53-95489acfc511/prometheus-operator-admission-webhook/0.log" Jul 07 18:09:09.790864 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.790827 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5"] Jul 07 18:09:09.795825 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.795796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:09.805150 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.805124 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5"] Jul 07 18:09:09.927399 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.927336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-lib-modules\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:09.927636 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.927414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-proc\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:09.927636 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.927486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-podres\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:09.927636 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.927530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhwm\" (UniqueName: \"kubernetes.io/projected/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-kube-api-access-tbhwm\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:09.927636 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:09.927624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-sys\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-proc\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028669 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-podres\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhwm\" (UniqueName: \"kubernetes.io/projected/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-kube-api-access-tbhwm\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-sys\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-proc\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028820 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-sys\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-podres\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-lib-modules\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.028988 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.028981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-lib-modules\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.040194 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.040158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhwm\" (UniqueName: \"kubernetes.io/projected/790d757b-c9ad-49b3-86d4-50e80c1ac3a8-kube-api-access-tbhwm\") pod \"perf-node-gather-daemonset-vtgk5\" (UID: \"790d757b-c9ad-49b3-86d4-50e80c1ac3a8\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.112334 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.112222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.302970 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.302936 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5"] Jul 07 18:09:10.305887 ip-10-0-137-197 kubenswrapper[2569]: W0707 18:09:10.305849 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod790d757b_c9ad_49b3_86d4_50e80c1ac3a8.slice/crio-046ede62b1f3962d65c3a2d2025b98c2593da0fc0d0e24eacfae2c9a4c891c2a WatchSource:0}: Error finding container 046ede62b1f3962d65c3a2d2025b98c2593da0fc0d0e24eacfae2c9a4c891c2a: Status 404 returned error can't find the container with id 046ede62b1f3962d65c3a2d2025b98c2593da0fc0d0e24eacfae2c9a4c891c2a Jul 07 18:09:10.818756 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.818715 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" event={"ID":"790d757b-c9ad-49b3-86d4-50e80c1ac3a8","Type":"ContainerStarted","Data":"423e19f3262cf5600e70ca4876c1ab4fb2885c82a8f98bc8b26773a9c80389a1"} Jul 07 18:09:10.818959 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.818779 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" event={"ID":"790d757b-c9ad-49b3-86d4-50e80c1ac3a8","Type":"ContainerStarted","Data":"046ede62b1f3962d65c3a2d2025b98c2593da0fc0d0e24eacfae2c9a4c891c2a"} Jul 07 18:09:10.818959 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.818892 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:10.825492 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.825464 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-ch6fp_4926c65f-8c54-4aa5-9ea1-b9e59b52b44b/networking-console-plugin/0.log" Jul 07 18:09:10.841300 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:10.841232 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" podStartSLOduration=1.841211497 podStartE2EDuration="1.841211497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:09:10.838374202 +0000 UTC m=+2815.958578533" watchObservedRunningTime="2026-07-07 18:09:10.841211497 +0000 UTC m=+2815.961415829" Jul 07 18:09:11.483191 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:11.483160 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/1.log" Jul 07 18:09:11.496546 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:11.496516 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2ffz_4d370278-be42-4bb8-a24e-1e035a961a3d/console-operator/2.log" Jul 07 18:09:12.066361 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:12.066328 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6ddfbbf57d-7zhwz_6bad37ed-24d3-4542-9f25-c90237db54cb/console/0.log" Jul 07 18:09:12.136243 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:12.136210 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-zpbpz_8912dfbf-eba0-43ed-895a-06b623bacc7e/download-server/0.log" Jul 07 18:09:12.823330 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:12.823273 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-ddhbx_13577d5c-0545-4095-8369-48b4d6557a61/volume-data-source-validator/0.log" Jul 07 18:09:14.099018 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:14.098990 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxjt7_c1ed0db7-bbde-4b36-9d77-85a4b6328a24/dns/0.log" Jul 07 18:09:14.126104 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:14.126071 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxjt7_c1ed0db7-bbde-4b36-9d77-85a4b6328a24/kube-rbac-proxy/0.log" Jul 07 18:09:14.349950 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:14.349868 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qkd9s_595c820e-0463-4011-851d-f3dea5f189e6/dns-node-resolver/0.log" Jul 07 18:09:15.073872 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:15.073767 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hnzsh_75eaed74-372b-4f2d-a80e-bfb1b0398601/node-ca/0.log" Jul 07 18:09:16.152906 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:16.152866 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rlb6w_4f7a0f87-b8ec-405b-9e62-c84aed63a0cf/istio-proxy/0.log" Jul 07 18:09:16.637538 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:16.637500 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-784748979c-znp9d_bb2371ec-c2d8-4094-a9e4-939d0601b6d5/router/0.log" Jul 07 18:09:16.838914 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:16.838883 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-vtgk5" Jul 07 18:09:17.480476 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:17.480445 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqgft_5391d841-23bf-4e93-848e-b2788bc05d76/serve-healthcheck-canary/0.log" Jul 07 18:09:18.278193 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:18.278128 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-85ggm_79a1fb9f-b7f2-4692-9b45-c9871bc7eb58/insights-operator/0.log" Jul 07 18:09:18.282413 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:18.282388 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-85ggm_79a1fb9f-b7f2-4692-9b45-c9871bc7eb58/insights-operator/1.log" Jul 07 18:09:18.311702 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:18.311671 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jl4g_5e3a6b56-8f0f-4368-9afc-d3d485c54416/kube-rbac-proxy/0.log" Jul 07 18:09:18.342854 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:18.342827 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jl4g_5e3a6b56-8f0f-4368-9afc-d3d485c54416/exporter/0.log" Jul 07 18:09:18.375496 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:18.375460 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jl4g_5e3a6b56-8f0f-4368-9afc-d3d485c54416/extractor/0.log" Jul 07 18:09:20.747328 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:20.747271 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rlb6w_4f7a0f87-b8ec-405b-9e62-c84aed63a0cf/istio-proxy/0.log" Jul 07 18:09:21.133532 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:21.133445 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-784748979c-znp9d_bb2371ec-c2d8-4094-a9e4-939d0601b6d5/router/0.log" Jul 07 18:09:21.963925 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:21.963840 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-k8qxq_402b619f-1f41-4c4b-935d-522b81b6e4b5/manager/0.log" Jul 07 18:09:22.024662 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:22.024630 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-k8qxq_402b619f-1f41-4c4b-935d-522b81b6e4b5/copy-manifests/0.log" Jul 07 18:09:22.402028 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:22.401999 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rlb6w_4f7a0f87-b8ec-405b-9e62-c84aed63a0cf/istio-proxy/0.log" Jul 07 18:09:22.765325 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:22.765213 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-784748979c-znp9d_bb2371ec-c2d8-4094-a9e4-939d0601b6d5/router/0.log" Jul 07 18:09:23.510177 ip-10-0-137-197 kubenswrapper[2569]: I0707 18:09:23.510141 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-686dcd8c67-5j8wn_1292a3e0-bd1e-4fcb-b923-a88813ab6427/manager/0.log"